Re: [openstack-dev] [Marconi] Kafka support and high throughput
On 04/06/14 14:58 +, Hochmuth, Roland M wrote: Hi Flavio, In your discussions around a developing Kafka plugin for Marconi would that be potentially be done by adding a Kafka transport to oslo.messaging? That is something that I'm very interested in for the monitoring as a service project I'm working on. Hey Roland, No, oslo.messaging is a different project and Marconi doesn't rely on it. Cheers, Flavio Thanks --Roland On 6/4/14, 3:06 AM, "Flavio Percoco" wrote: On 02/06/14 07:52 -0700, Keith Newstadt wrote: Thanks for the responses Flavio, Roland. Some background on why I'm asking: we're using Kafka as the message queue for a stream processing service we're building, which we're delivering to our internal customers as a service along with OpenStack. We're considering building a high throughput ingest API to get the clients' data streams into the stream processing service. It occurs to me that this API is simply a messaging API, and so I'm wondering if we should consider building this high throughput API as part of the Marconi project. Has this topic come up in the Marconi team's discussions, and would it fit into the vision of the Marconi roadmap? Yes it has and I'm happy to see this coming up in the ML, thanks. Some things that we're considering in order to have a more flexible architecture that will support a higher throughput are: - Queue Flavors (Terrible name). This is for marconi what flavors are for Nova. It basically defines a set of properties that will belong to a queue. Some of those properties may be related to the messages lifetime or the storage capabilities (in-memory, freaking fast, durable, etc). This is yet to be done. - 2 new drivers (AMQP, redis). The former adds support to brokers and the later to well, redis, which brings in support for in-memory queues. Work In Progress. - A new transport. This is something we've discussed but we haven't reached an agreement yet on when this should be done nor what it should be based on. The gist of this feature is adding support for another protocol that can serve Marconi's API alongside the HTTP one. We've considered TCP and websocket so far. The former is perfect for lower level communications without the HTTP overhead whereas the later is useful for web apps. That said. A Kafka plugin is something we heard a lot about at the summit and we've discussed it a bit. I'd love to see that happening as an external plugin for now. There's no need to wait for the rest to happen. I'm more than happy to help with guidance and support on the repo creation, driver structure etc. Cheers, Flavio Thanks, Keith Newstadt keith_newst...@symantec.com @knewstadt Date: Sun, 1 Jun 2014 15:01:40 + From: "Hochmuth, Roland M" To: OpenStack List Subject: Re: [openstack-dev] [Marconi] Kafka support and high throughput Message-ID: Content-Type: text/plain; charset="us-ascii" There are some folks in HP evaluating different messaging technologies for Marconi, such as RabbitMQ and Kafka. I'll ping them and maybe they can share some information. On a related note, the Monitoring as a Service solution we are working on uses Kafka. This was just open-sourced at, https://github.com/hpcloud-mon, and will be moving over to StackForge starting next week. The architecture is at, https://github.com/hpcloud-mon/mon-arch. I haven't really looked at Marconi. If you are interested in throughput, low latency, durability, scale and fault-tolerance Kafka seems like a great choice. It has been also pointed out from various sources that possibly Kafka could be another oslo.messaging transport. Are you looking into that as that would be very interesting to me and something that is on my task list that I haven't gotten to yet. On 5/30/14, 7:03 AM, "Keith Newstadt" wrote: Has anyone given thought to using Kafka to back Marconi? And has there been discussion about adding high throughput APIs to Marconi. We're looking at providing Kafka as a messaging service for our customers, in a scenario where throughput is a priority. We've had good luck using both streaming HTTP interfaces and long poll interfaces to get high throughput for other web services we've built. Would this use case be appropriate in the context of the Marconi roadmap? Thanks, Keith Newstadt keith_newst...@symantec.com Keith Newstadt Cloud Services Architect Cloud Platform Engineering Symantec Corporation www.symantec.com Office: (781) 530-2299 Mobile: (617) 513-1321 Email: keith_newst...@symantec.com Twitter: @knewstadt This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privi
Re: [openstack-dev] [Marconi] Kafka support and high throughput
Hi Flavio, In your discussions around a developing Kafka plugin for Marconi would that be potentially be done by adding a Kafka transport to oslo.messaging? That is something that I'm very interested in for the monitoring as a service project I'm working on. Thanks --Roland On 6/4/14, 3:06 AM, "Flavio Percoco" wrote: >On 02/06/14 07:52 -0700, Keith Newstadt wrote: >>Thanks for the responses Flavio, Roland. >> >>Some background on why I'm asking: we're using Kafka as the message >>queue for a stream processing service we're building, which we're >>delivering to our internal customers as a service along with OpenStack. >>We're considering building a high throughput ingest API to get the >>clients' data streams into the stream processing service. It occurs to >>me that this API is simply a messaging API, and so I'm wondering if we >>should consider building this high throughput API as part of the Marconi >>project. >> >>Has this topic come up in the Marconi team's discussions, and would it >>fit into the vision of the Marconi roadmap? > >Yes it has and I'm happy to see this coming up in the ML, thanks. > >Some things that we're considering in order to have a more flexible >architecture that will support a higher throughput are: > >- Queue Flavors (Terrible name). This is for marconi what flavors are > for Nova. It basically defines a set of properties that will belong > to a queue. Some of those properties may be related to the messages > lifetime or the storage capabilities (in-memory, freaking fast, > durable, etc). This is yet to be done. > >- 2 new drivers (AMQP, redis). The former adds support to brokers and > the later to well, redis, which brings in support for in-memory > queues. Work In Progress. > >- A new transport. This is something we've discussed but we haven't > reached an agreement yet on when this should be done nor what it > should be based on. The gist of this feature is adding support for > another protocol that can serve Marconi's API alongside the HTTP > one. We've considered TCP and websocket so far. The former is > perfect for lower level communications without the HTTP overhead > whereas the later is useful for web apps. > >That said. A Kafka plugin is something we heard a lot about at the >summit and we've discussed it a bit. I'd love to see that happening as >an external plugin for now. There's no need to wait for the rest to >happen. > >I'm more than happy to help with guidance and support on the repo >creation, driver structure etc. > >Cheers, >Flavio > >> >>Thanks, >>Keith Newstadt >>keith_newst...@symantec.com >>@knewstadt >> >> >>Date: Sun, 1 Jun 2014 15:01:40 + >>From: "Hochmuth, Roland M" >>To: OpenStack List >>Subject: Re: [openstack-dev] [Marconi] Kafka support and high >> throughput >>Message-ID: >>Content-Type: text/plain; charset="us-ascii" >> >>There are some folks in HP evaluating different messaging technologies >>for >>Marconi, such as RabbitMQ and Kafka. I'll ping them and maybe they can >>share >>some information. >> >>On a related note, the Monitoring as a Service solution we are working >>on uses Kafka. This was just open-sourced at, >>https://github.com/hpcloud-mon, >>and will be moving over to StackForge starting next week. The >>architecture >>is at, >>https://github.com/hpcloud-mon/mon-arch. >> >>I haven't really looked at Marconi. If you are interested in >>throughput, low latency, durability, scale and fault-tolerance Kafka >>seems like a great choice. >> >>It has been also pointed out from various sources that possibly Kafka >>could be another oslo.messaging transport. Are you looking into that as >>that would be very interesting to me and something that is on my task >>list that I haven't gotten to yet. >> >> >>On 5/30/14, 7:03 AM, "Keith Newstadt" >>wrote: >> >>>Has anyone given thought to using Kafka to back Marconi? And has there >>>been discussion about adding high throughput APIs to Marconi. >>> >>>We're looking at providing Kafka as a messaging service for our >>>customers, in a scenario where throughput is a priority. We've had good >>>luck using both streaming HTTP interfaces and long poll interfaces to >>>get >>>high throughput for other web services we've built. Would this use case >>>be appropriate in the context of the Marconi ro
Re: [openstack-dev] [Marconi] Kafka support and high throughput
On 02/06/14 07:52 -0700, Keith Newstadt wrote: Thanks for the responses Flavio, Roland. Some background on why I'm asking: we're using Kafka as the message queue for a stream processing service we're building, which we're delivering to our internal customers as a service along with OpenStack. We're considering building a high throughput ingest API to get the clients' data streams into the stream processing service. It occurs to me that this API is simply a messaging API, and so I'm wondering if we should consider building this high throughput API as part of the Marconi project. Has this topic come up in the Marconi team's discussions, and would it fit into the vision of the Marconi roadmap? Yes it has and I'm happy to see this coming up in the ML, thanks. Some things that we're considering in order to have a more flexible architecture that will support a higher throughput are: - Queue Flavors (Terrible name). This is for marconi what flavors are for Nova. It basically defines a set of properties that will belong to a queue. Some of those properties may be related to the messages lifetime or the storage capabilities (in-memory, freaking fast, durable, etc). This is yet to be done. - 2 new drivers (AMQP, redis). The former adds support to brokers and the later to well, redis, which brings in support for in-memory queues. Work In Progress. - A new transport. This is something we've discussed but we haven't reached an agreement yet on when this should be done nor what it should be based on. The gist of this feature is adding support for another protocol that can serve Marconi's API alongside the HTTP one. We've considered TCP and websocket so far. The former is perfect for lower level communications without the HTTP overhead whereas the later is useful for web apps. That said. A Kafka plugin is something we heard a lot about at the summit and we've discussed it a bit. I'd love to see that happening as an external plugin for now. There's no need to wait for the rest to happen. I'm more than happy to help with guidance and support on the repo creation, driver structure etc. Cheers, Flavio Thanks, Keith Newstadt keith_newst...@symantec.com @knewstadt Date: Sun, 1 Jun 2014 15:01:40 +0000 From: "Hochmuth, Roland M" To: OpenStack List Subject: Re: [openstack-dev] [Marconi] Kafka support and high throughput Message-ID: Content-Type: text/plain; charset="us-ascii" There are some folks in HP evaluating different messaging technologies for Marconi, such as RabbitMQ and Kafka. I'll ping them and maybe they can share some information. On a related note, the Monitoring as a Service solution we are working on uses Kafka. This was just open-sourced at, https://github.com/hpcloud-mon, and will be moving over to StackForge starting next week. The architecture is at, https://github.com/hpcloud-mon/mon-arch. I haven't really looked at Marconi. If you are interested in throughput, low latency, durability, scale and fault-tolerance Kafka seems like a great choice. It has been also pointed out from various sources that possibly Kafka could be another oslo.messaging transport. Are you looking into that as that would be very interesting to me and something that is on my task list that I haven't gotten to yet. On 5/30/14, 7:03 AM, "Keith Newstadt" wrote: Has anyone given thought to using Kafka to back Marconi? And has there been discussion about adding high throughput APIs to Marconi. We're looking at providing Kafka as a messaging service for our customers, in a scenario where throughput is a priority. We've had good luck using both streaming HTTP interfaces and long poll interfaces to get high throughput for other web services we've built. Would this use case be appropriate in the context of the Marconi roadmap? Thanks, Keith Newstadt keith_newst...@symantec.com Keith Newstadt Cloud Services Architect Cloud Platform Engineering Symantec Corporation www.symantec.com Office: (781) 530-2299 Mobile: (617) 513-1321 Email: keith_newst...@symantec.com Twitter: @knewstadt This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or may constitute as attorney work product. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, notify us immediately by telephone and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. _
Re: [openstack-dev] [Marconi] Kafka support and high throughput
Keith, Have you put HTTP protocol in front of Kafka, or are you using Kafka¹s native protocol? Can you also expand a little on your performance requirements? What does ³high throughput² mean to you in terms of the messaging patterns (# of producers and consumers, # of queues and queue partitions, message size, desired throughput)? Thanks, Tomasz Janczuk @tjanczuk HP On 5/31/14, 11:56 PM, "Flavio Percoco" wrote: >On 30/05/14 06:03 -0700, Keith Newstadt wrote: >>Has anyone given thought to using Kafka to back Marconi? And has there >>been discussion about adding high throughput APIs to Marconi. >> >>We're looking at providing Kafka as a messaging service for our >>customers, in a scenario where throughput is a priority. We've had good >>luck using both streaming HTTP interfaces and long poll interfaces to >>get high throughput for other web services we've built. Would this use >>case be appropriate in the context of the Marconi roadmap? > > >Hi, > >Kafka would be a good store to back Marconi with. We've had some >feedback from the community w.r.t Kafka and there seems to be lot of >interest on it. The team is not currently targeting it but we could >probably do something after the J release. > >That said, Marconi's plugin architecture allows people to create >third-party drivers and obviously use them. It'd be really nice to see >some work going on that area as an external driver. > >Thanks, >Flavio > >-- >@flaper87 >Flavio Percoco >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Marconi] Kafka support and high throughput
Thanks for the responses Flavio, Roland. Some background on why I'm asking: we're using Kafka as the message queue for a stream processing service we're building, which we're delivering to our internal customers as a service along with OpenStack. We're considering building a high throughput ingest API to get the clients' data streams into the stream processing service. It occurs to me that this API is simply a messaging API, and so I'm wondering if we should consider building this high throughput API as part of the Marconi project. Has this topic come up in the Marconi team's discussions, and would it fit into the vision of the Marconi roadmap? Thanks, Keith Newstadt keith_newst...@symantec.com @knewstadt Date: Sun, 1 Jun 2014 15:01:40 + From: "Hochmuth, Roland M" To: OpenStack List Subject: Re: [openstack-dev] [Marconi] Kafka support and high throughput Message-ID: Content-Type: text/plain; charset="us-ascii" There are some folks in HP evaluating different messaging technologies for Marconi, such as RabbitMQ and Kafka. I'll ping them and maybe they can share some information. On a related note, the Monitoring as a Service solution we are working on uses Kafka. This was just open-sourced at, https://github.com/hpcloud-mon, and will be moving over to StackForge starting next week. The architecture is at, https://github.com/hpcloud-mon/mon-arch. I haven't really looked at Marconi. If you are interested in throughput, low latency, durability, scale and fault-tolerance Kafka seems like a great choice. It has been also pointed out from various sources that possibly Kafka could be another oslo.messaging transport. Are you looking into that as that would be very interesting to me and something that is on my task list that I haven't gotten to yet. On 5/30/14, 7:03 AM, "Keith Newstadt" wrote: >Has anyone given thought to using Kafka to back Marconi? And has there >been discussion about adding high throughput APIs to Marconi. > >We're looking at providing Kafka as a messaging service for our >customers, in a scenario where throughput is a priority. We've had good >luck using both streaming HTTP interfaces and long poll interfaces to get >high throughput for other web services we've built. Would this use case >be appropriate in the context of the Marconi roadmap? > >Thanks, >Keith Newstadt >keith_newst...@symantec.com > Keith Newstadt Cloud Services Architect Cloud Platform Engineering Symantec Corporation www.symantec.com Office: (781) 530-2299 Mobile: (617) 513-1321 Email: keith_newst...@symantec.com Twitter: @knewstadt This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or may constitute as attorney work product. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, notify us immediately by telephone and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Marconi] Kafka support and high throughput
There are some folks in HP evaluating different messaging technologies for Marconi, such as RabbitMQ and Kafka. I'll ping them and maybe they can share some information. On a related note, the Monitoring as a Service solution we are working on uses Kafka. This was just open-sourced at, https://github.com/hpcloud-mon, and will be moving over to StackForge starting next week. The architecture is at, https://github.com/hpcloud-mon/mon-arch. I haven't really looked at Marconi. If you are interested in throughput, low latency, durability, scale and fault-tolerance Kafka seems like a great choice. It has been also pointed out from various sources that possibly Kafka could be another oslo.messaging transport. Are you looking into that as that would be very interesting to me and something that is on my task list that I haven't gotten to yet. On 5/30/14, 7:03 AM, "Keith Newstadt" wrote: >Has anyone given thought to using Kafka to back Marconi? And has there >been discussion about adding high throughput APIs to Marconi. > >We're looking at providing Kafka as a messaging service for our >customers, in a scenario where throughput is a priority. We've had good >luck using both streaming HTTP interfaces and long poll interfaces to get >high throughput for other web services we've built. Would this use case >be appropriate in the context of the Marconi roadmap? > >Thanks, >Keith Newstadt >keith_newst...@symantec.com > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Marconi] Kafka support and high throughput
On 30/05/14 06:03 -0700, Keith Newstadt wrote: Has anyone given thought to using Kafka to back Marconi? And has there been discussion about adding high throughput APIs to Marconi. We're looking at providing Kafka as a messaging service for our customers, in a scenario where throughput is a priority. We've had good luck using both streaming HTTP interfaces and long poll interfaces to get high throughput for other web services we've built. Would this use case be appropriate in the context of the Marconi roadmap? Hi, Kafka would be a good store to back Marconi with. We've had some feedback from the community w.r.t Kafka and there seems to be lot of interest on it. The team is not currently targeting it but we could probably do something after the J release. That said, Marconi's plugin architecture allows people to create third-party drivers and obviously use them. It'd be really nice to see some work going on that area as an external driver. Thanks, Flavio -- @flaper87 Flavio Percoco pgp60Zq_sr0E2.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Marconi] Kafka support and high throughput
Has anyone given thought to using Kafka to back Marconi? And has there been discussion about adding high throughput APIs to Marconi. We're looking at providing Kafka as a messaging service for our customers, in a scenario where throughput is a priority. We've had good luck using both streaming HTTP interfaces and long poll interfaces to get high throughput for other web services we've built. Would this use case be appropriate in the context of the Marconi roadmap? Thanks, Keith Newstadt keith_newst...@symantec.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
