Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On Fri, Oct 3, 2014 at 4:05 AM, Soren Hansen so...@linux2go.dk wrote: I'm sorry about my slow responses. For some reason, gmail didn't think this was an important e-mail :( 2014-09-30 18:41 GMT+02:00 Jay Pipes jaypi...@gmail.com: On 09/30/2014 08:03 AM, Soren Hansen wrote: 2014-09-12 1:05 GMT+02:00 Jay Pipes jaypi...@gmail.com: How would I go about getting the associated fixed IPs for a network? The query to get associated fixed IPs for a network [1] in Nova looks like this: SELECT fip.address, fip.instance_uuid, [...] AND fip.instance_uuid IS NOT NULL AND i.host = :host would I have a Riak container for virtual_interfaces that would also have instance information, network information, fixed_ip information? How would I accomplish the query against a derived table that gets the minimum virtual interface ID for each instance UUID? What's a minimum virtual interface ID? Anyway, I think Clint answered this quite well. I've said it before, and I'll say it again. In Nova at least, the SQL schema is complex because the problem domain is complex. That means lots of relations, lots of JOINs, and that means the best way to query for that data is via an RDBMS. [...] I don't think relying on a central data store is in any conceivable way appropriate for a project like OpenStack. Least of all Nova. I don't see how we can build a highly available, distributed service on top of a centralized data store like MySQL. [...] I don't disagree with anything you say above. At all. Really? How can you agree that we can't build a highly available, distributed service on top of a centralized data store like MySQL while also saying that the best way to handle data in Nova is in an RDBMS? For complex control plane software like Nova, though, an RDBMS is the best tool for the job given the current lay of the land in open source data storage solutions matched with Nova's complex query and transactional requirements. What transactional requirements? https://github.com/openstack/nova/blob/stable/icehouse/nova/db/sqlalchemy/api.py#L1654 When you delete an instance, you don't want the delete to just stop half-way through the transaction and leave around a bunch of orphaned children. Similarly, when you reserve something, it helps to not have a half-finished state change that you need to go clean up if something goes boom. Looking at that particular example, it's about deleting an instance and all its associated metadata. As we established earlier, these are things that would just be in the same key as the instance itself, so it'd just be a single key that would get deleted. Easy. That said, there will certainly be situations where there'll be a need for some sort of anti-entropy mechanism. It just so happens that those situations already exist. We're dealing with about a complex distributed system. We're kidding ourselves if we think that any kind of consistency is guaranteed, just because our data store favours consistency over availability. I apologize if I'm missing something, but doesn't denormalization to add join support put the same value in many places, such that an update to that value is no longer a single atomic transaction? This would appear to counteract the requirement for strong consistency. If updating a single value is atomic (as in Riak's consistent mode) then it might be possible to construct a way to make multiple updates appear atomic, but it would add many more transactions and many more quorum checks, which would reduce performance to a crawl. I also don't really see how a NoSQL system in strong consistency mode is any different from running MySQL with galera in its failure modes. The requirement for quorum makes the addition of nodes increase the potential latency of writes (and reads in some cases) so having large scale doesn't grant much benefit, if any. Quorum will also prevent nodes on the wrong side of a partition from being able to access system state (or it will give them stale state, which is probably just as bad in our case). I think your goal of having state management that's able to handle network partitions is a good one, but I don't think the solution is as simple as swapping out where the state is stored. Maybe in some cases like split-racks the system needs to react to a network partition by forming its own independent cell with its own state storage, and when the network heals it then merges back into the other cluster cleanly? That would be very difficult to implement, but fun (for some definition of fun). As a thought experiment, a while ago I considered what would happen if instead of using a central store, I put a sqlite database behind every daemon and allowed them to query each other for the data they needed, and cluster if needed (using raft). Services like nova-scheduler need strong consistency and would have to cluster to perform their role, but services like nova-compute would
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
2014-10-03 9:00 GMT+02:00 Michael Chapman wop...@gmail.com: On Fri, Oct 3, 2014 at 4:05 AM, Soren Hansen so...@linux2go.dk wrote: That said, there will certainly be situations where there'll be a need for some sort of anti-entropy mechanism. It just so happens that those situations already exist. We're dealing with about a complex distributed system. We're kidding ourselves if we think that any kind of consistency is guaranteed, just because our data store favours consistency over availability. I apologize if I'm missing something, but doesn't denormalization to add join support put the same value in many places, such that an update to that value is no longer a single atomic transaction? Yes. This would appear to counteract the requirement for strong consistency. What requirement for strong consistency? If updating a single value is atomic (as in Riak's consistent mode) Admittedly, I'm not 100% up-to-date on Riak, but last I looked, there wasn't any consistent mode. However, when writing a value, you can specify that you want all (or a quorum of) replicas to be written to disk before you get a succesful response. However, this does not imply transactional support. In other words, if one of the writes fail, it doesn't get rolled back on the other nodes. You just don't get a succesful response. I also don't really see how a NoSQL system in strong consistency mode is any different from running MySQL with galera in its failure modes. I agree. I never meant to imply that we should run anything in strong consistency mode. There might be a few operations that require strong consistency, but they should be exceptional. Quotas sound like a good example. The requirement for quorum makes the addition of nodes increase the potential latency of writes (and reads in some cases) so having large scale doesn't grant much benefit, if any. I agree about the requirement for quorum having those effects (also for e.g. Galera). I think you are missing my point, though. My concerns are not whether MySQL can handle the data volume of a large scale OpenStack deployment. I'm sure it can. Without even breaking a sweat. MySQL has been used in countless deployments to handle data sets vastly bigger than what we're dealing with. My concern is reliability. Quorum will also prevent nodes on the wrong side of a partition from being able to access system state (or it will give them stale state, which is probably just as bad in our case). This problem exists today. Suppose you have a 5 node Galera cluster. Would you refuse reads on the wrong side of the partition to avoid providing stale data? With e.g. Riak it's perfectly possible to accept both reads and writes on both sides of the partition. No matter what we do, we need accept the fact that when we handle the data, it is by definition out of date. It can have changed the millisecond after we read it from there and started using it. I think your goal of having state management that's able to handle network partitions is a good one, but I don't think the solution is as simple as swapping out where the state is stored. It kinda is, and it kinda isn't. I never meant to suggest that just replacing the datastore would solve everything. We need to carefully look at our use of the data from the datastore and consider the impact of eventual consistency on this use. On the other hand, as I just mentioned above, this is a problem that exists right now, today. We're just ignoring it, because we happen to have a consistent datastore. Maybe in some cases like split-racks the system needs to react to a network partition by forming its own independent cell with its own state storage, and when the network heals it then merges back into the other cluster cleanly? That would be very difficult to implement, but fun (for some definition of fun). Fun, but possible. Riak was designed for this. With an RDBMS I don't even know how to begin solving something like that. As a thought experiment, a while ago I considered what would happen if instead of using a central store, I put a sqlite database behind every daemon and allowed them to query each other for the data they needed, and cluster if needed (using raft). Services like nova-scheduler need strong consistency No, it doesn't. :) and would have to cluster to perform their role, but services like nova-compute would simply need to store the data concerning the resources they are responsible for. This follows the 'place state at the edge' kind of design principles that have been discussed in various circles. It falls down in a number of pretty obvious ways, and ultimately it would require more work than I am able to put in, but I mention it because perhaps it provides you with food for thought. Yeah, a million distributed consistent databases do not a single distributed, eventually consistent database make :) -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
I'm sorry about my slow responses. For some reason, gmail didn't think this was an important e-mail :( 2014-09-30 18:41 GMT+02:00 Jay Pipes jaypi...@gmail.com: On 09/30/2014 08:03 AM, Soren Hansen wrote: 2014-09-12 1:05 GMT+02:00 Jay Pipes jaypi...@gmail.com: How would I go about getting the associated fixed IPs for a network? The query to get associated fixed IPs for a network [1] in Nova looks like this: SELECT fip.address, fip.instance_uuid, [...] AND fip.instance_uuid IS NOT NULL AND i.host = :host would I have a Riak container for virtual_interfaces that would also have instance information, network information, fixed_ip information? How would I accomplish the query against a derived table that gets the minimum virtual interface ID for each instance UUID? What's a minimum virtual interface ID? Anyway, I think Clint answered this quite well. I've said it before, and I'll say it again. In Nova at least, the SQL schema is complex because the problem domain is complex. That means lots of relations, lots of JOINs, and that means the best way to query for that data is via an RDBMS. [...] I don't think relying on a central data store is in any conceivable way appropriate for a project like OpenStack. Least of all Nova. I don't see how we can build a highly available, distributed service on top of a centralized data store like MySQL. [...] I don't disagree with anything you say above. At all. Really? How can you agree that we can't build a highly available, distributed service on top of a centralized data store like MySQL while also saying that the best way to handle data in Nova is in an RDBMS? For complex control plane software like Nova, though, an RDBMS is the best tool for the job given the current lay of the land in open source data storage solutions matched with Nova's complex query and transactional requirements. What transactional requirements? https://github.com/openstack/nova/blob/stable/icehouse/nova/db/sqlalchemy/api.py#L1654 When you delete an instance, you don't want the delete to just stop half-way through the transaction and leave around a bunch of orphaned children. Similarly, when you reserve something, it helps to not have a half-finished state change that you need to go clean up if something goes boom. Looking at that particular example, it's about deleting an instance and all its associated metadata. As we established earlier, these are things that would just be in the same key as the instance itself, so it'd just be a single key that would get deleted. Easy. That said, there will certainly be situations where there'll be a need for some sort of anti-entropy mechanism. It just so happens that those situations already exist. We're dealing with about a complex distributed system. We're kidding ourselves if we think that any kind of consistency is guaranteed, just because our data store favours consistency over availability. https://github.com/openstack/nova/blob/stable/icehouse/nova/db/sqlalchemy/api.py#L3054 Sure, quotas will require stronger consistency. Any NoSQL data store worth its salt gives you primitives to implement that. Folks in these other programs have actually, you know, thought about these kinds of things and had serious discussions about alternatives. It would be nice to have someone acknowledge that instead of snarky comments implying everyone else has it wrong. I'm terribly sorry, but repeating over and over that an RDBMS is the best tool without further qualification than Nova's data model is really complex reads *exactly* like a snarky comment implying everyone else has it wrong. Sorry if I sound snarky. I thought your blog post was the definition of snark. I don't see the relevance of the tone of my blog post? You say it would be nice if people did something other than offer snarky comments implying everyone else has it wrong. I'm just pointing out that such requests ring really hollow when put forth in the very e-mail where you snarkily tell everyone else that they have it wrong. Since you did bring up my blog post, I really am astounded you find it snarky. It was intended to be constructive and forward looking. The first one in the series, perhaps, but certainly not the one linked in this thread. Perhaps I need to take writing classes. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/30/2014 02:03 PM, Soren Hansen wrote: 2014-09-12 1:05 GMT+02:00 Jay Pipes jaypi...@gmail.com: If Nova was to take Soren's advice and implement its data-access layer on top of Cassandra or Riak, we would just end up re-inventing SQL Joins in Python-land. I may very well be wrong(!), but this statement makes it sound like you've never used e.g. Riak. Or, if you have, not done so in the way it's supposed to be used. If you embrace an alternative way of storing your data, you wouldn't just blindly create a container for each table in your RDBMS. For example: In Nova's SQL-based datastore we have a table for security groups and another for security group rules. Rows in the security group rules table have a foreign key referencing the security group to which they belong. In a datastore like Riak, you could have a security group container where each value contains not just the security group information, but also all the security group rules. No joins in Python-land necessary. I've said it before, and I'll say it again. In Nova at least, the SQL schema is complex because the problem domain is complex. That means lots of relations, lots of JOINs, and that means the best way to query for that data is via an RDBMS. I was really hoping you could be more specific than best/most appropriate so that we could have a focused discussion. I don't think relying on a central data store is in any conceivable way appropriate for a project like OpenStack. Least of all Nova. I don't see how we can build a highly available, distributed service on top of a centralized data store like MySQL. Coming from Skype background I can assure your that you definitely can, depending on your needs (and our experiments with e.g. MongoDB ended very badly: it just died under IO loads, that our PostgreSQL treated like normal). I mean, that's complex topic and I see a lot of people switching to NoSQL and a lot of people switching from. NoSQL is not a silver bullet for scalability. Just my 0.5. /me disappears again Tens or hundreds of thousands of nodes, spread across many, many racks and datacentre halls are going to experience connectivity problems[1]. This means that some percentage of your infrastructure (possibly many thousands of nodes, affecting many, many thousands of customers) will find certain functionality not working on account of your datastore not being reachable from the part of the control plane they're attempting to use (or possibly only being able to read from it). I say over and over again that people should own their own uptime. Expect things to fail all the time. Do whatever you need to do to ensure your service keeps working even when something goes wrong. Of course this applies to our customers too. Even if we take the greatest care to avoid downtime, customers should spread their workloads across multiple availability zones and/or regions and probably even multiple cloud providers. Their service towards their users is their responsibility. However, our service towards our users is our responsibility. We should take the greatest care to avoid having internal problems affect our users. Building a massively distributed system like Nova on top of a centralized data store is practically a guarantee of the opposite. For complex control plane software like Nova, though, an RDBMS is the best tool for the job given the current lay of the land in open source data storage solutions matched with Nova's complex query and transactional requirements. What transactional requirements? Folks in these other programs have actually, you know, thought about these kinds of things and had serious discussions about alternatives. It would be nice to have someone acknowledge that instead of snarky comments implying everyone else has it wrong. I'm terribly sorry, but repeating over and over that an RDBMS is the best tool without further qualification than Nova's data model is really complex reads *exactly* like a snarky comment implying everyone else has it wrong. [1]: http://aphyr.com/posts/288-the-network-is-reliable ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
2014-09-12 1:05 GMT+02:00 Jay Pipes jaypi...@gmail.com: If Nova was to take Soren's advice and implement its data-access layer on top of Cassandra or Riak, we would just end up re-inventing SQL Joins in Python-land. I may very well be wrong(!), but this statement makes it sound like you've never used e.g. Riak. Or, if you have, not done so in the way it's supposed to be used. If you embrace an alternative way of storing your data, you wouldn't just blindly create a container for each table in your RDBMS. For example: In Nova's SQL-based datastore we have a table for security groups and another for security group rules. Rows in the security group rules table have a foreign key referencing the security group to which they belong. In a datastore like Riak, you could have a security group container where each value contains not just the security group information, but also all the security group rules. No joins in Python-land necessary. I've said it before, and I'll say it again. In Nova at least, the SQL schema is complex because the problem domain is complex. That means lots of relations, lots of JOINs, and that means the best way to query for that data is via an RDBMS. I was really hoping you could be more specific than best/most appropriate so that we could have a focused discussion. I don't think relying on a central data store is in any conceivable way appropriate for a project like OpenStack. Least of all Nova. I don't see how we can build a highly available, distributed service on top of a centralized data store like MySQL. Tens or hundreds of thousands of nodes, spread across many, many racks and datacentre halls are going to experience connectivity problems[1]. This means that some percentage of your infrastructure (possibly many thousands of nodes, affecting many, many thousands of customers) will find certain functionality not working on account of your datastore not being reachable from the part of the control plane they're attempting to use (or possibly only being able to read from it). I say over and over again that people should own their own uptime. Expect things to fail all the time. Do whatever you need to do to ensure your service keeps working even when something goes wrong. Of course this applies to our customers too. Even if we take the greatest care to avoid downtime, customers should spread their workloads across multiple availability zones and/or regions and probably even multiple cloud providers. Their service towards their users is their responsibility. However, our service towards our users is our responsibility. We should take the greatest care to avoid having internal problems affect our users. Building a massively distributed system like Nova on top of a centralized data store is practically a guarantee of the opposite. For complex control plane software like Nova, though, an RDBMS is the best tool for the job given the current lay of the land in open source data storage solutions matched with Nova's complex query and transactional requirements. What transactional requirements? Folks in these other programs have actually, you know, thought about these kinds of things and had serious discussions about alternatives. It would be nice to have someone acknowledge that instead of snarky comments implying everyone else has it wrong. I'm terribly sorry, but repeating over and over that an RDBMS is the best tool without further qualification than Nova's data model is really complex reads *exactly* like a snarky comment implying everyone else has it wrong. [1]: http://aphyr.com/posts/288-the-network-is-reliable -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/30/2014 08:03 AM, Soren Hansen wrote: 2014-09-12 1:05 GMT+02:00 Jay Pipes jaypi...@gmail.com: If Nova was to take Soren's advice and implement its data-access layer on top of Cassandra or Riak, we would just end up re-inventing SQL Joins in Python-land. I may very well be wrong(!), but this statement makes it sound like you've never used e.g. Riak. Or, if you have, not done so in the way it's supposed to be used. If you embrace an alternative way of storing your data, you wouldn't just blindly create a container for each table in your RDBMS. For example: In Nova's SQL-based datastore we have a table for security groups and another for security group rules. Rows in the security group rules table have a foreign key referencing the security group to which they belong. In a datastore like Riak, you could have a security group container where each value contains not just the security group information, but also all the security group rules. No joins in Python-land necessary. OK, that's all fine for a simple one-to-many relation. How would I go about getting the associated fixed IPs for a network? The query to get associated fixed IPs for a network [1] in Nova looks like this: SELECT fip.address, fip.instance_uuid, fip.network_id, fip.virtual_interface_id, vif.address, i.hostname, i.updated_at, i.created_at, fip.allocated, fip.leased, vif2.id FROM fixed_ips fip LEFT JOIN virtual_interfaces vif ON vif.id = fip.virtual_interface_id AND vif.deleted = 0 LEFT JOIN instances i ON fip.instance_uuid = i.uuid AND i.deleted = 0 LEFT JOIN ( SELECT MIN(vi.id) AS id, vi.instance_uuid FROM virtual_interfaces vi GROUP BY instance_uuid ) as vif2 WHERE fip.deleted = 0 AND fip.network_id = :network_id AND fip.virtual_interface_id IS NOT NULL AND fip.instance_uuid IS NOT NULL AND i.host = :host would I have a Riak container for virtual_interfaces that would also have instance information, network information, fixed_ip information? How would I accomplish the query against a derived table that gets the minimum virtual interface ID for each instance UUID? More than likely, I would end up having to put a bunch of indexes and relations into my Riak containers and structures just so I could do queries like the above. Failing that, I'd need to do multiple queries to multiple Riak containers and then join the resulting projection in memory, in Python. And that is why I say you will just end up implementing joins in Python. A relational database was built for the above types of queries, and that's why I said it's the best tool for the job *in this specific case*. Now... that said... Is it possible to go through the Nova schema and identify mini-schemas that could be pulled out of the RDBMS and placed into Riak or Cassandra? Absolutely yes! The service group and compute node usage records are good candidates for that, in my opinion. With the nova.objects work that was completed over the last few cycles, we might actually now have the foundation in place to make doing this a reality. I welcome your contributions in this area. [1] https://github.com/openstack/nova/blob/stable/icehouse/nova/db/sqlalchemy/api.py#L2608 I've said it before, and I'll say it again. In Nova at least, the SQL schema is complex because the problem domain is complex. That means lots of relations, lots of JOINs, and that means the best way to query for that data is via an RDBMS. I was really hoping you could be more specific than best/most appropriate so that we could have a focused discussion. I don't think relying on a central data store is in any conceivable way appropriate for a project like OpenStack. Least of all Nova. I don't see how we can build a highly available, distributed service on top of a centralized data store like MySQL. Tens or hundreds of thousands of nodes, spread across many, many racks and datacentre halls are going to experience connectivity problems[1]. This means that some percentage of your infrastructure (possibly many thousands of nodes, affecting many, many thousands of customers) will find certain functionality not working on account of your datastore not being reachable from the part of the control plane they're attempting to use (or possibly only being able to read from it). I say over and over again that people should own their own uptime. Expect things to fail all the time. Do whatever you need to do to ensure your service keeps working even when something goes wrong. Of course this applies to our customers too. Even if we take the greatest care to avoid downtime, customers should spread their workloads across multiple availability zones and/or regions and probably even multiple cloud providers. Their service towards their users is their responsibility. However, our service towards our users is our responsibility. We should take the greatest care to avoid having internal problems affect our users. Building a massively distributed system like Nova on top of a
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
Excerpts from Jay Pipes's message of 2014-09-30 09:41:29 -0700: On 09/30/2014 08:03 AM, Soren Hansen wrote: 2014-09-12 1:05 GMT+02:00 Jay Pipes jaypi...@gmail.com: If Nova was to take Soren's advice and implement its data-access layer on top of Cassandra or Riak, we would just end up re-inventing SQL Joins in Python-land. I may very well be wrong(!), but this statement makes it sound like you've never used e.g. Riak. Or, if you have, not done so in the way it's supposed to be used. If you embrace an alternative way of storing your data, you wouldn't just blindly create a container for each table in your RDBMS. For example: In Nova's SQL-based datastore we have a table for security groups and another for security group rules. Rows in the security group rules table have a foreign key referencing the security group to which they belong. In a datastore like Riak, you could have a security group container where each value contains not just the security group information, but also all the security group rules. No joins in Python-land necessary. OK, that's all fine for a simple one-to-many relation. How would I go about getting the associated fixed IPs for a network? The query to get associated fixed IPs for a network [1] in Nova looks like this: SELECT fip.address, fip.instance_uuid, fip.network_id, fip.virtual_interface_id, vif.address, i.hostname, i.updated_at, i.created_at, fip.allocated, fip.leased, vif2.id FROM fixed_ips fip LEFT JOIN virtual_interfaces vif ON vif.id = fip.virtual_interface_id AND vif.deleted = 0 LEFT JOIN instances i ON fip.instance_uuid = i.uuid AND i.deleted = 0 LEFT JOIN ( SELECT MIN(vi.id) AS id, vi.instance_uuid FROM virtual_interfaces vi GROUP BY instance_uuid ) as vif2 WHERE fip.deleted = 0 AND fip.network_id = :network_id AND fip.virtual_interface_id IS NOT NULL AND fip.instance_uuid IS NOT NULL AND i.host = :host You and I both know that this query is not something we want to be running a lot. In the past when I've had systems where something like the above needed to be run a lot, I created materialized views for it and made access to this information asynchronous because relying on this in real-time is _extremely expensive_. This is where a massively scalable but somewhat dumb database shines because you can materialize the view into it at a much faster pace and with more scaled-out workers so that you now have fine grained resource allocations for scaling the responsiveness of this view. would I have a Riak container for virtual_interfaces that would also have instance information, network information, fixed_ip information? How would I accomplish the query against a derived table that gets the minimum virtual interface ID for each instance UUID? Yes you'd have a bunch of duplicated information everywhere. Asynchronous denormalizing is the new join. The only worry I'd have is what accidental contracts have we made that require bits of information to appear all at once. :-P More than likely, I would end up having to put a bunch of indexes and relations into my Riak containers and structures just so I could do queries like the above. Failing that, I'd need to do multiple queries to multiple Riak containers and then join the resulting projection in memory, in Python. And that is why I say you will just end up implementing joins in Python. My experience has been that you have a map/reduce cluster somewhere churning through updates to pre-compute things for slow queries and you do any realtime duplication at write time. The idea is to create documents that are themselves useful without the need of the other documents. Where this approach suffers is when your documents all have so many copies of data that you end up writing 10x more than you read. That is a rare thing, and most certainly would not happen with Nova's data, which mostly would have duplicates of lots of UUID's and IP addresses. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
Excerpts from Jay Pipes's message of 2014-09-30 09:41:29 -0700: A relational database was built for the above types of queries, and that's why I said it's the best tool for the job *in this specific case*. Now... that said... Is it possible to go through the Nova schema and identify mini-schemas that could be pulled out of the RDBMS and placed into Riak or Cassandra? Absolutely yes! The service group and compute node usage records are good candidates for that, in my opinion. With the nova.objects work that was completed over the last few cycles, we might actually now have the foundation in place to make doing this a reality. I welcome your contributions in this area. I forgot to say, +1 here. :) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
2014-09-26 17:11 GMT+02:00 Jay Pipes jaypi...@gmail.com: On 09/26/2014 06:45 AM, Soren Hansen wrote: Define best. best == most appropriate. #copout -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
2014-09-12 1:05 GMT+02:00 Jay Pipes jaypi...@gmail.com: If Nova was to take Soren's advice and implement its data-access layer on top of Cassandra or Riak, we would just end up re-inventing SQL Joins in Python-land. I've said it before, and I'll say it again. In Nova at least, the SQL schema is complex because the problem domain is complex. That means lots of relations, lots of JOINs, and that means the best way to query for that data is via an RDBMS. Define best. /Soren ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/26/2014 06:45 AM, Soren Hansen wrote: 2014-09-12 1:05 GMT+02:00 Jay Pipes jaypi...@gmail.com: If Nova was to take Soren's advice and implement its data-access layer on top of Cassandra or Riak, we would just end up re-inventing SQL Joins in Python-land. I've said it before, and I'll say it again. In Nova at least, the SQL schema is complex because the problem domain is complex. That means lots of relations, lots of JOINs, and that means the best way to query for that data is via an RDBMS. Define best. best == most appropriate. -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/12/2014 03:29 AM, Clint Byrum wrote: Excerpts from Zane Bitter's message of 2014-09-11 15:21:26 -0700: On 09/09/14 19:56, Clint Byrum wrote: Excerpts from Samuel Merritt's message of 2014-09-09 16:12:09 -0700: On 9/9/14, 12:03 PM, Monty Taylor wrote: On 09/04/2014 01:30 AM, Clint Byrum wrote: Excerpts from Flavio Percoco's message of 2014-09-04 00:08:47 -0700: Greetings, Last Tuesday the TC held the first graduation review for Zaqar. During the meeting some concerns arose. I've listed those concerns below with some comments hoping that it will help starting a discussion before the next meeting. In addition, I've added some comments about the project stability at the bottom and an etherpad link pointing to a list of use cases for Zaqar. Hi Flavio. This was an interesting read. As somebody whose attention has recently been drawn to Zaqar, I am quite interested in seeing it graduate. # Concerns - Concern on operational burden of requiring NoSQL deploy expertise to the mix of openstack operational skills For those of you not familiar with Zaqar, it currently supports 2 nosql drivers - MongoDB and Redis - and those are the only 2 drivers it supports for now. This will require operators willing to use Zaqar to maintain a new (?) NoSQL technology in their system. Before expressing our thoughts on this matter, let me say that: 1. By removing the SQLAlchemy driver, we basically removed the chance for operators to use an already deployed OpenStack-technology 2. Zaqar won't be backed by any AMQP based messaging technology for now. Here's[0] a summary of the research the team (mostly done by Victoria) did during Juno 3. We (OpenStack) used to require Redis for the zmq matchmaker 4. We (OpenStack) also use memcached for caching and as the oslo caching lib becomes available - or a wrapper on top of dogpile.cache - Redis may be used in place of memcached in more and more deployments. 5. Ceilometer's recommended storage driver is still MongoDB, although Ceilometer has now support for sqlalchemy. (Please correct me if I'm wrong). That being said, it's obvious we already, to some extent, promote some NoSQL technologies. However, for the sake of the discussion, lets assume we don't. I truly believe, with my OpenStack (not Zaqar's) hat on, that we can't keep avoiding these technologies. NoSQL technologies have been around for years and we should be prepared - including OpenStack operators - to support these technologies. Not every tool is good for all tasks - one of the reasons we removed the sqlalchemy driver in the first place - therefore it's impossible to keep an homogeneous environment for all services. I whole heartedly agree that non traditional storage technologies that are becoming mainstream are good candidates for use cases where SQL based storage gets in the way. I wish there wasn't so much FUD (warranted or not) about MongoDB, but that is the reality we live in. With this, I'm not suggesting to ignore the risks and the extra burden this adds but, instead of attempting to avoid it completely by not evolving the stack of services we provide, we should probably work on defining a reasonable subset of NoSQL services we are OK with supporting. This will help making the burden smaller and it'll give operators the option to choose. [0] http://blog.flaper87.com/post/marconi-amqp-see-you-later/ - Concern on should we really reinvent a queue system rather than piggyback on one As mentioned in the meeting on Tuesday, Zaqar is not reinventing message brokers. Zaqar provides a service akin to SQS from AWS with an OpenStack flavor on top. [0] I think Zaqar is more like SMTP and IMAP than AMQP. You're not really trying to connect two processes in real time. You're trying to do fully asynchronous messaging with fully randomized access to any message. Perhaps somebody should explore whether the approaches taken by large scale IMAP providers could be applied to Zaqar. Anyway, I can't imagine writing a system to intentionally use the semantics of IMAP and SMTP. I'd be very interested in seeing actual use cases for it, apologies if those have been posted before. It seems like you're EITHER describing something called XMPP that has at least one open source scalable backend called ejabberd. OR, you've actually hit the nail on the head with bringing up SMTP and IMAP but for some reason that feels strange. SMTP and IMAP already implement every feature you've described, as well as retries/failover/HA and a fully end to end secure transport (if installed properly) If you don't actually set them up to run as a public messaging interface but just as a cloud-local exchange, then you could get by with very low overhead for a massive throughput - it can very easily be run on a single machine for Sean's simplicity, and could just as easily be scaled out using well known techniques for public cloud
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/12/2014 12:14 AM, Zane Bitter wrote: On 09/09/14 15:03, Monty Taylor wrote: On 09/04/2014 01:30 AM, Clint Byrum wrote: Excerpts from Flavio Percoco's message of 2014-09-04 00:08:47 -0700: Greetings, Last Tuesday the TC held the first graduation review for Zaqar. During the meeting some concerns arose. I've listed those concerns below with some comments hoping that it will help starting a discussion before the next meeting. In addition, I've added some comments about the project stability at the bottom and an etherpad link pointing to a list of use cases for Zaqar. Hi Flavio. This was an interesting read. As somebody whose attention has recently been drawn to Zaqar, I am quite interested in seeing it graduate. # Concerns - Concern on operational burden of requiring NoSQL deploy expertise to the mix of openstack operational skills For those of you not familiar with Zaqar, it currently supports 2 nosql drivers - MongoDB and Redis - and those are the only 2 drivers it supports for now. This will require operators willing to use Zaqar to maintain a new (?) NoSQL technology in their system. Before expressing our thoughts on this matter, let me say that: 1. By removing the SQLAlchemy driver, we basically removed the chance for operators to use an already deployed OpenStack-technology 2. Zaqar won't be backed by any AMQP based messaging technology for now. Here's[0] a summary of the research the team (mostly done by Victoria) did during Juno 3. We (OpenStack) used to require Redis for the zmq matchmaker 4. We (OpenStack) also use memcached for caching and as the oslo caching lib becomes available - or a wrapper on top of dogpile.cache - Redis may be used in place of memcached in more and more deployments. 5. Ceilometer's recommended storage driver is still MongoDB, although Ceilometer has now support for sqlalchemy. (Please correct me if I'm wrong). That being said, it's obvious we already, to some extent, promote some NoSQL technologies. However, for the sake of the discussion, lets assume we don't. I truly believe, with my OpenStack (not Zaqar's) hat on, that we can't keep avoiding these technologies. NoSQL technologies have been around for years and we should be prepared - including OpenStack operators - to support these technologies. Not every tool is good for all tasks - one of the reasons we removed the sqlalchemy driver in the first place - therefore it's impossible to keep an homogeneous environment for all services. I whole heartedly agree that non traditional storage technologies that are becoming mainstream are good candidates for use cases where SQL based storage gets in the way. I wish there wasn't so much FUD (warranted or not) about MongoDB, but that is the reality we live in. With this, I'm not suggesting to ignore the risks and the extra burden this adds but, instead of attempting to avoid it completely by not evolving the stack of services we provide, we should probably work on defining a reasonable subset of NoSQL services we are OK with supporting. This will help making the burden smaller and it'll give operators the option to choose. [0] http://blog.flaper87.com/post/marconi-amqp-see-you-later/ - Concern on should we really reinvent a queue system rather than piggyback on one As mentioned in the meeting on Tuesday, Zaqar is not reinventing message brokers. Zaqar provides a service akin to SQS from AWS with an OpenStack flavor on top. [0] I think Zaqar is more like SMTP and IMAP than AMQP. You're not really trying to connect two processes in real time. You're trying to do fully asynchronous messaging with fully randomized access to any message. Perhaps somebody should explore whether the approaches taken by large scale IMAP providers could be applied to Zaqar. Anyway, I can't imagine writing a system to intentionally use the semantics of IMAP and SMTP. I'd be very interested in seeing actual use cases for it, apologies if those have been posted before. It seems like you're EITHER describing something called XMPP that has at least one open source scalable backend called ejabberd. OR, you've actually hit the nail on the head with bringing up SMTP and IMAP but for some reason that feels strange. SMTP and IMAP already implement every feature you've described, as well as retries/failover/HA and a fully end to end secure transport (if installed properly) If you don't actually set them up to run as a public messaging interface but just as a cloud-local exchange, then you could get by with very low overhead for a massive throughput - it can very easily be run on a single machine for Sean's simplicity, and could just as easily be scaled out using well known techniques for public cloud sized deployments? So why not use existing daemons that do this? You could still use the REST API you've got, but instead of writing it to a mongo backend and trying to implement
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
Flavio Percoco wrote: On 09/12/2014 12:14 AM, Zane Bitter wrote: The final question is the one of arbitrary access to messages in the queue (or queue if you prefer). Flavio indicated that this effectively came for free with their implementation of Pub-Sub. IMHO it is unnecessary and limits the choice of potential back ends in the future. I would personally be +1 on removing it from the v2 API, and also +1 on the v2 API shipping in Kilo so that as few new adopters as possible get stuck with the limited choices of back-end. I hope that would resolve Clint's concerns that we need a separate, light-weight queue system; I personally don't believe we need two projects, even though I agree that all of the use cases I personally care about could probably be satisfied without Pub-Sub. Right, being able to support other backends is one of the reasons we're looking forward to remove the support for arbitrary access to messages. As of now, the plan is to remove that endpoint unless a very good use case comes up that makes supporting other backends not worth it, which I doubt. The feedback from Zaqar's early adopters is that the endpoint is indeed not useful. Thanks Zane, that was indeed useful. I agree with you it would be better to avoid needing 2 separate projects for such close use cases. Let's assume we remove arbitrary access to messages in v2. When you say it would remove limits on the choice of potential backends, does that mean we could have a pure queue backend (like RabbitMQ), at least in theory ? Would a ZaqarV2 address all of Clint and Devananda's concerns about queue semantics ? If yes, then the graduation question becomes, how likely is that work to be completed early enough in Kilo. If it's a no-brainer and takes a week to sort out, I think we could approve Zaqar's Kilo graduation, even if that stretches the no major API rewrite planned requirement. But if we think this needs careful discussion so that the v2 API design (and backend support) satisfies the widest set of users, then incubating for another cycle while v2 is implemented seems like the right course of action. We shouldn't graduate if there is any risk we would end up with ZaqarV1 in Kilo, and then have to deprecate it for n cycles just because it was shipped in the official release and therefore inherits its API deprecation rules. Regards, -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On Wed, 2014-09-10 at 12:46 -0700, Monty Taylor wrote: On 09/09/2014 07:04 PM, Samuel Merritt wrote: On 9/9/14, 4:47 PM, Devananda van der Veen wrote: The questions now before us are: - should OpenStack include, in the integrated release, a messaging-as-a-service component? I certainly think so. I've worked on a few reasonable-scale web applications, and they all followed the same pattern: HTTP app servers serving requests quickly, background workers for long-running tasks, and some sort of durable message-broker/queue-server thing for conveying work from the first to the second. A quick straw poll of my nearby coworkers shows that every non-trivial web application that they've worked on in the last decade follows the same pattern. While not *every* application needs such a thing, web apps are quite common these days, and Zaqar satisfies one of their big requirements. Not only that, it does so in a way that requires much less babysitting than run-your-own-broker does. Right. But here's the thing. What you just described is what we all thought zaqar was aiming to be in the beginning. We did not think it was a GOOD implementation of that, so while we agreed that it would be useful to have one of those, we were not crazy about the implementation. Those generalizations are uncomfortably sweeping. What Samuel just described is one of the messaging patterns that Zaqar implements and some (members of the TC?) believed that this messaging pattern was the only pattern that Zaqar aimed to implement. Some (members of the TC?) formed strong, negative opinions about how this messaging pattern was implemented, but some/all of those same people agreed a messaging API implementing those semantics would be a useful thing to have. Mark. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/12/2014 11:36 AM, Thierry Carrez wrote: Flavio Percoco wrote: On 09/12/2014 12:14 AM, Zane Bitter wrote: The final question is the one of arbitrary access to messages in the queue (or queue if you prefer). Flavio indicated that this effectively came for free with their implementation of Pub-Sub. IMHO it is unnecessary and limits the choice of potential back ends in the future. I would personally be +1 on removing it from the v2 API, and also +1 on the v2 API shipping in Kilo so that as few new adopters as possible get stuck with the limited choices of back-end. I hope that would resolve Clint's concerns that we need a separate, light-weight queue system; I personally don't believe we need two projects, even though I agree that all of the use cases I personally care about could probably be satisfied without Pub-Sub. Right, being able to support other backends is one of the reasons we're looking forward to remove the support for arbitrary access to messages. As of now, the plan is to remove that endpoint unless a very good use case comes up that makes supporting other backends not worth it, which I doubt. The feedback from Zaqar's early adopters is that the endpoint is indeed not useful. Thanks Zane, that was indeed useful. I agree with you it would be better to avoid needing 2 separate projects for such close use cases. +1 Let's assume we remove arbitrary access to messages in v2. When you say it would remove limits on the choice of potential backends, does that mean we could have a pure queue backend (like RabbitMQ), at least in theory ? Would a ZaqarV2 address all of Clint and Devananda's concerns about queue semantics ? If yes, then the graduation question becomes, how likely is that work to be completed early enough in Kilo. If it's a no-brainer and takes a week to sort out, I think we could approve Zaqar's Kilo graduation, even if that stretches the no major API rewrite planned requirement. Let me break the above down into several points so we can discuss them separately: - Removing that endpoint won't take more than a week. It's an API change and it won't affect the existing storage drivers. - Removing that endpoint will certainly make the adoption of other messaging technologies easier but there are other things to consider besides that specific endpoint (some of them were stated here[0]). In any case, removing the endpoint definitely makes it easier. - Besides the random access to messages, I'm not clear what other concerns there are with regards the current semantics. It'd be nice if we could recollect them in this section and discuss them. I took a look at the other emails in this thread and it seems to me that the concerns that have been raised are more oriented to the project scope and use-cases. I also looked at the meeting logs again[1] and the only concern related to the semantics I found is about the `get-message-by-id` endpoint. Please, correct me if I'm wrong. [0] http://blog.flaper87.com/post/marconi-amqp-see-you-later/ [1] http://eavesdrop.openstack.org/meetings/tc/2014/tc.2014-09-09-20.01.log.html Flavio But if we think this needs careful discussion so that the v2 API design (and backend support) satisfies the widest set of users, then incubating for another cycle while v2 is implemented seems like the right course of action. We shouldn't graduate if there is any risk we would end up with ZaqarV1 in Kilo, and then have to deprecate it for n cycles just because it was shipped in the official release and therefore inherits its API deprecation rules. Regards, -- @flaper87 Flavio Percoco ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 11/09/14 19:05, Jay Pipes wrote: On 09/11/2014 04:09 PM, Zane Bitter wrote: Swift is the current exception here, but one could argue, and people have[2], that Swift is also the only project that actually conforms to our stated design tenets for OpenStack. I'd struggle to tell the Zaqar folks they've done the Wrong Thing... especially when abandoning the RDBMS driver was done largely at the direction of the TC iirc. snip [2] http://blog.linux2go.dk/2013/08/30/openstack-design-tenets-part-2/ No offense to Soren, who wrote some interesting and poignant things, nor to the Swift developers, who continue to produce excellent work, but Swift is object storage. It is a data plane system with a small API surface, a very limited functional domain, and a small, inflexible storage schema (which is perfectly fine for its use cases). It's needs for a relational database are nearly non-existent. It replicates a SQLite database around using rsync [1]. Try doing that with a schema of any complexity and you will quickly find the limitations of such a strategy. If Nova was to take Soren's advice and implement its data-access layer on top of Cassandra or Riak, we would just end up re-inventing SQL Joins in Python-land. I've said it before, and I'll say it again. In Nova at least, the SQL schema is complex because the problem domain is complex. That means lots of relations, lots of JOINs, and that means the best way to query for that data is via an RDBMS. And I say that knowing just how *poor* some of the queries are in Nova! I wasn't trying to suggest that Nova should change (if there was another project I had in mind while reading that it would have been Heat, not Nova). My point was that it's understandable that Zaqar, which is *also* a data-plane service with a small API surface and a limited functional domain, doesn't have the same architecture as Nova (just as Swift doesn't) and that it's probably counter-productive to force it into that architecture purely because a bunch of other things use it. For projects like Swift, Zaqar, even Keystone, Glance and Cinder, a non-RDBMS solution might be a perfectly reasonable solution for the underlying data storage and access layer (and for the record, I never said that Zaqar should or should not use an RDBMS for its storage). For complex control plane software like Nova, though, an RDBMS is the best tool for the job given the current lay of the land in open source data storage solutions matched with Nova's complex query and transactional requirements. +1 Folks in these other programs have actually, you know, thought about these kinds of things and had serious discussions about alternatives. It would be nice to have someone acknowledge that instead of snarky comments implying everyone else has it wrong. I didn't mean to imply that anybody else has it wrong (although FWIW I do think that Heat probably has it wrong), and I apologise to anyone who interpreted it that way. Going back in my hole, -jay No! Let's talk about Zaqar :) cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 12/09/14 04:50, Flavio Percoco wrote: On 09/12/2014 12:14 AM, Zane Bitter wrote: However, Zaqar also supports the Pub-Sub model of messaging. I believe, but would like Flavio to confirm, that this is what is meant when the Zaqar team say that Zaqar is about messaging in general and not just queuing. That is to say, it is possible for multiple consumers to intentionally consume the same message, with each maintaining its own pointer in the queue. (Another way to think of this is that messages can be multicast to multiple virtual queues, with data de-duplication between them.) To a relative novice in the field like me, the difference between this and queuing sounds pretty academic :P. Call it what you will, it seems like a reasonable thing to implement to me. Correct, this and other messaging patterns supported by Zaqar make it a messaging service, which as Gordon mentioned in another email is just a more generic term. Messages are the most important resource in Zaqar and providing good, common and scalable patterns to access those messages is what we strive for in Zaqar's API. Thanks Flavio! I think we are more or less on the same page :) Maybe you could clarify what the other messaging patterns are exactly, since that seems to be one of the points of confusion/contention. cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/12/2014 09:50 AM, Flavio Percoco wrote: Zaqar supports once and only once delivery. For the transfer from Zaqar to consumers it does (providing the claim id can be recovered). For transfer from producers to Zaqar I believe it is more limited. If the connection to Zaqar fails during a post, the sender can't tell whether the message was successfully enqueued or not. It could try to determine this is by browsing the entire queue looking for a matching body. However thatt would be awkward and in any case the absence of the message could mean that it wasn't enqueued or that it was already consumed and deleted. One way of handling this is to have the post return a unique url to which the message(s) are put or posted. The sender can then repost (re-put) to this in the event of failure and the server can determine whether it already processed the publications. Alternatively the client can be required to generate a unique id on which the server can de-deduplicate. The ActiveMQ REST interface supports both of these approaches: http://activemq.apache.org/restful-queue.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/12/2014 01:56 PM, Flavio Percoco wrote: On 09/12/2014 11:36 AM, Thierry Carrez wrote: Flavio Percoco wrote: On 09/12/2014 12:14 AM, Zane Bitter wrote: The final question is the one of arbitrary access to messages in the queue (or queue if you prefer). Flavio indicated that this effectively came for free with their implementation of Pub-Sub. IMHO it is unnecessary and limits the choice of potential back ends in the future. I would personally be +1 on removing it from the v2 API, and also +1 on the v2 API shipping in Kilo so that as few new adopters as possible get stuck with the limited choices of back-end. I hope that would resolve Clint's concerns that we need a separate, light-weight queue system; I personally don't believe we need two projects, even though I agree that all of the use cases I personally care about could probably be satisfied without Pub-Sub. Right, being able to support other backends is one of the reasons we're looking forward to remove the support for arbitrary access to messages. As of now, the plan is to remove that endpoint unless a very good use case comes up that makes supporting other backends not worth it, which I doubt. The feedback from Zaqar's early adopters is that the endpoint is indeed not useful. Thanks Zane, that was indeed useful. I agree with you it would be better to avoid needing 2 separate projects for such close use cases. +1 Let's assume we remove arbitrary access to messages in v2. When you say it would remove limits on the choice of potential backends, does that mean we could have a pure queue backend (like RabbitMQ), at least in theory ? Would a ZaqarV2 address all of Clint and Devananda's concerns about queue semantics ? If yes, then the graduation question becomes, how likely is that work to be completed early enough in Kilo. If it's a no-brainer and takes a week to sort out, I think we could approve Zaqar's Kilo graduation, even if that stretches the no major API rewrite planned requirement. Let me break the above down into several points so we can discuss them separately: - Removing that endpoint won't take more than a week. It's an API change and it won't affect the existing storage drivers. For the sake of discussion and to provide more info on this point, I've done this (there are still some tests to clean-up but that's basically all that's required): https://review.openstack.org/#/c/121141/ Flavio - Removing that endpoint will certainly make the adoption of other messaging technologies easier but there are other things to consider besides that specific endpoint (some of them were stated here[0]). In any case, removing the endpoint definitely makes it easier. - Besides the random access to messages, I'm not clear what other concerns there are with regards the current semantics. It'd be nice if we could recollect them in this section and discuss them. I took a look at the other emails in this thread and it seems to me that the concerns that have been raised are more oriented to the project scope and use-cases. I also looked at the meeting logs again[1] and the only concern related to the semantics I found is about the `get-message-by-id` endpoint. Please, correct me if I'm wrong. [0] http://blog.flaper87.com/post/marconi-amqp-see-you-later/ [1] http://eavesdrop.openstack.org/meetings/tc/2014/tc.2014-09-09-20.01.log.html Flavio But if we think this needs careful discussion so that the v2 API design (and backend support) satisfies the widest set of users, then incubating for another cycle while v2 is implemented seems like the right course of action. We shouldn't graduate if there is any risk we would end up with ZaqarV1 in Kilo, and then have to deprecate it for n cycles just because it was shipped in the official release and therefore inherits its API deprecation rules. Regards, -- @flaper87 Flavio Percoco ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
Excerpts from Flavio Percoco's message of 2014-09-12 00:22:35 -0700: On 09/12/2014 03:29 AM, Clint Byrum wrote: Excerpts from Zane Bitter's message of 2014-09-11 15:21:26 -0700: On 09/09/14 19:56, Clint Byrum wrote: Excerpts from Samuel Merritt's message of 2014-09-09 16:12:09 -0700: On 9/9/14, 12:03 PM, Monty Taylor wrote: On 09/04/2014 01:30 AM, Clint Byrum wrote: Excerpts from Flavio Percoco's message of 2014-09-04 00:08:47 -0700: Greetings, Last Tuesday the TC held the first graduation review for Zaqar. During the meeting some concerns arose. I've listed those concerns below with some comments hoping that it will help starting a discussion before the next meeting. In addition, I've added some comments about the project stability at the bottom and an etherpad link pointing to a list of use cases for Zaqar. Hi Flavio. This was an interesting read. As somebody whose attention has recently been drawn to Zaqar, I am quite interested in seeing it graduate. # Concerns - Concern on operational burden of requiring NoSQL deploy expertise to the mix of openstack operational skills For those of you not familiar with Zaqar, it currently supports 2 nosql drivers - MongoDB and Redis - and those are the only 2 drivers it supports for now. This will require operators willing to use Zaqar to maintain a new (?) NoSQL technology in their system. Before expressing our thoughts on this matter, let me say that: 1. By removing the SQLAlchemy driver, we basically removed the chance for operators to use an already deployed OpenStack-technology 2. Zaqar won't be backed by any AMQP based messaging technology for now. Here's[0] a summary of the research the team (mostly done by Victoria) did during Juno 3. We (OpenStack) used to require Redis for the zmq matchmaker 4. We (OpenStack) also use memcached for caching and as the oslo caching lib becomes available - or a wrapper on top of dogpile.cache - Redis may be used in place of memcached in more and more deployments. 5. Ceilometer's recommended storage driver is still MongoDB, although Ceilometer has now support for sqlalchemy. (Please correct me if I'm wrong). That being said, it's obvious we already, to some extent, promote some NoSQL technologies. However, for the sake of the discussion, lets assume we don't. I truly believe, with my OpenStack (not Zaqar's) hat on, that we can't keep avoiding these technologies. NoSQL technologies have been around for years and we should be prepared - including OpenStack operators - to support these technologies. Not every tool is good for all tasks - one of the reasons we removed the sqlalchemy driver in the first place - therefore it's impossible to keep an homogeneous environment for all services. I whole heartedly agree that non traditional storage technologies that are becoming mainstream are good candidates for use cases where SQL based storage gets in the way. I wish there wasn't so much FUD (warranted or not) about MongoDB, but that is the reality we live in. With this, I'm not suggesting to ignore the risks and the extra burden this adds but, instead of attempting to avoid it completely by not evolving the stack of services we provide, we should probably work on defining a reasonable subset of NoSQL services we are OK with supporting. This will help making the burden smaller and it'll give operators the option to choose. [0] http://blog.flaper87.com/post/marconi-amqp-see-you-later/ - Concern on should we really reinvent a queue system rather than piggyback on one As mentioned in the meeting on Tuesday, Zaqar is not reinventing message brokers. Zaqar provides a service akin to SQS from AWS with an OpenStack flavor on top. [0] I think Zaqar is more like SMTP and IMAP than AMQP. You're not really trying to connect two processes in real time. You're trying to do fully asynchronous messaging with fully randomized access to any message. Perhaps somebody should explore whether the approaches taken by large scale IMAP providers could be applied to Zaqar. Anyway, I can't imagine writing a system to intentionally use the semantics of IMAP and SMTP. I'd be very interested in seeing actual use cases for it, apologies if those have been posted before. It seems like you're EITHER describing something called XMPP that has at least one open source scalable backend called ejabberd. OR, you've actually hit the nail on the head with bringing up SMTP and IMAP but for some reason that feels strange. SMTP and IMAP already implement every feature you've described, as well as retries/failover/HA and a fully end to end secure transport (if installed properly) If you don't actually set them up to run as a public messaging interface but just as a cloud-local exchange, then you could get by with very low
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 04/09/14 08:14, Sean Dague wrote: I've been one of the consistent voices concerned about a hard requirement on adding NoSQL into the mix. So I'll explain that thinking a bit more. I feel like when the TC makes an integration decision previously this has been about evaluating the project applying for integration, and if they met some specific criteria they were told about some time in the past. I think that's the wrong approach. It's a locally optimized approach that fails to ask the more interesting question. Is OpenStack better as a whole if this is a mandatory component of OpenStack? Better being defined as technically better (more features, less janky code work arounds, less unexpected behavior from the stack). Better from the sense of easier or harder to run an actual cloud by our Operators (taking into account what kinds of moving parts they are now expected to manage). Better from the sense of a better user experience in interacting with OpenStack as whole. Better from a sense that the OpenStack release will experience less bugs, less unexpected cross project interactions, an a greater overall feel of consistency so that the OpenStack API feels like one thing. https://dague.net/2014/08/26/openstack-as-layers/ I don't want to get off-topic here, but I want to state before this becomes the de-facto starting point for a layering discussion that I don't accept this model at all. It is not based on any analysis whatsoever but appears to be entirely arbitrary - a collection of individual prejudices arranged visually. On a hopefully more constructive note, I believe there are at least two analyses that _would_ produce interesting data here: 1) Examine the dependencies, both hard and optional, between projects and enumerate the things you lose when ignoring each optional one. 2) Analyse projects based on the type of user consuming the service - e.g. Nova is mostly used (directly or indirectly via e.g. Heat and/or Horizon) by actual, corporeal persons, while Zaqar is used by both persons (to set up queues) and services (which actually send and receive messages) - of both OpenStack and applications. I believe, BTW that this analysis will uncover a lot of missing features in Keystone[1]. What you can _not_ produce is a linear model of the different types of clouds for different use cases, because different organisations have wildly differing needs. One of the interesting qualities of Layers 1 2 is they all follow an AMQP + RDBMS pattern (excepting swift). You can have a very effective IaaS out of that stack. They are the things that you can provide pretty solid integration testing on (and if you look at where everything stood before the new TC mandates on testing / upgrade that was basically what was getting integration tested). (Also note, I'll accept Barbican is probably in the wrong layer, and should be a Layer 2 service.) Swift is the current exception here, but one could argue, and people have[2], that Swift is also the only project that actually conforms to our stated design tenets for OpenStack. I'd struggle to tell the Zaqar folks they've done the Wrong Thing... especially when abandoning the RDBMS driver was done largely at the direction of the TC iirc. Speaking of Swift, I would really love to see it investigated as a potential storage backend for Zaqar. If it proves to have the right guarantees (and durability is the crucial one, so it sounds promising) then that has the potential to smooth over a lot of the deployment problem. While large shops can afford to have a dedicated team to figure out how to make mongo or redis HA, provide monitoring, have a DR plan for when a huricane requires them to flip datacenters, that basically means OpenStack heads further down the path of only for the big folks. I don't want OpenStack to be only for the big folks, I want OpenStack to be for all sized folks. I really do want to have all the local small colleges around here have OpenStack clouds, because it's something that people believe they can do and manage. I know the people that work in this places, they all come out to the LUG I run. We've talked about this. OpenStack is basically seen as too complex for them to use as it stands, and that pains me a ton. This is a great point, and one that we definitely have to keep in mind. It's also worth noting that small organisations also get the most benefit. Rather than having to stand up a cluster of reliable message brokers (large organisations are much more likely to need this kind of flexibility anyway) - potentially one cluster per application - they can have their IT department deploy e.g. a single Redis cluster and have messaging handled for every application in their cloud with all the benefits of multitenancy. Part of the move to the cloud is inevitably going to mean organisational changes in a lot of places, where the operations experts will increasingly focus on maintaining the cloud itself, rather
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/09/14 15:03, Monty Taylor wrote: On 09/04/2014 01:30 AM, Clint Byrum wrote: Excerpts from Flavio Percoco's message of 2014-09-04 00:08:47 -0700: Greetings, Last Tuesday the TC held the first graduation review for Zaqar. During the meeting some concerns arose. I've listed those concerns below with some comments hoping that it will help starting a discussion before the next meeting. In addition, I've added some comments about the project stability at the bottom and an etherpad link pointing to a list of use cases for Zaqar. Hi Flavio. This was an interesting read. As somebody whose attention has recently been drawn to Zaqar, I am quite interested in seeing it graduate. # Concerns - Concern on operational burden of requiring NoSQL deploy expertise to the mix of openstack operational skills For those of you not familiar with Zaqar, it currently supports 2 nosql drivers - MongoDB and Redis - and those are the only 2 drivers it supports for now. This will require operators willing to use Zaqar to maintain a new (?) NoSQL technology in their system. Before expressing our thoughts on this matter, let me say that: 1. By removing the SQLAlchemy driver, we basically removed the chance for operators to use an already deployed OpenStack-technology 2. Zaqar won't be backed by any AMQP based messaging technology for now. Here's[0] a summary of the research the team (mostly done by Victoria) did during Juno 3. We (OpenStack) used to require Redis for the zmq matchmaker 4. We (OpenStack) also use memcached for caching and as the oslo caching lib becomes available - or a wrapper on top of dogpile.cache - Redis may be used in place of memcached in more and more deployments. 5. Ceilometer's recommended storage driver is still MongoDB, although Ceilometer has now support for sqlalchemy. (Please correct me if I'm wrong). That being said, it's obvious we already, to some extent, promote some NoSQL technologies. However, for the sake of the discussion, lets assume we don't. I truly believe, with my OpenStack (not Zaqar's) hat on, that we can't keep avoiding these technologies. NoSQL technologies have been around for years and we should be prepared - including OpenStack operators - to support these technologies. Not every tool is good for all tasks - one of the reasons we removed the sqlalchemy driver in the first place - therefore it's impossible to keep an homogeneous environment for all services. I whole heartedly agree that non traditional storage technologies that are becoming mainstream are good candidates for use cases where SQL based storage gets in the way. I wish there wasn't so much FUD (warranted or not) about MongoDB, but that is the reality we live in. With this, I'm not suggesting to ignore the risks and the extra burden this adds but, instead of attempting to avoid it completely by not evolving the stack of services we provide, we should probably work on defining a reasonable subset of NoSQL services we are OK with supporting. This will help making the burden smaller and it'll give operators the option to choose. [0] http://blog.flaper87.com/post/marconi-amqp-see-you-later/ - Concern on should we really reinvent a queue system rather than piggyback on one As mentioned in the meeting on Tuesday, Zaqar is not reinventing message brokers. Zaqar provides a service akin to SQS from AWS with an OpenStack flavor on top. [0] I think Zaqar is more like SMTP and IMAP than AMQP. You're not really trying to connect two processes in real time. You're trying to do fully asynchronous messaging with fully randomized access to any message. Perhaps somebody should explore whether the approaches taken by large scale IMAP providers could be applied to Zaqar. Anyway, I can't imagine writing a system to intentionally use the semantics of IMAP and SMTP. I'd be very interested in seeing actual use cases for it, apologies if those have been posted before. It seems like you're EITHER describing something called XMPP that has at least one open source scalable backend called ejabberd. OR, you've actually hit the nail on the head with bringing up SMTP and IMAP but for some reason that feels strange. SMTP and IMAP already implement every feature you've described, as well as retries/failover/HA and a fully end to end secure transport (if installed properly) If you don't actually set them up to run as a public messaging interface but just as a cloud-local exchange, then you could get by with very low overhead for a massive throughput - it can very easily be run on a single machine for Sean's simplicity, and could just as easily be scaled out using well known techniques for public cloud sized deployments? So why not use existing daemons that do this? You could still use the REST API you've got, but instead of writing it to a mongo backend and trying to implement all of the things that already exist in SMTP/IMAP - you could just have them front to it. You could even bypass normal
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/09/14 19:56, Clint Byrum wrote: Excerpts from Samuel Merritt's message of 2014-09-09 16:12:09 -0700: On 9/9/14, 12:03 PM, Monty Taylor wrote: On 09/04/2014 01:30 AM, Clint Byrum wrote: Excerpts from Flavio Percoco's message of 2014-09-04 00:08:47 -0700: Greetings, Last Tuesday the TC held the first graduation review for Zaqar. During the meeting some concerns arose. I've listed those concerns below with some comments hoping that it will help starting a discussion before the next meeting. In addition, I've added some comments about the project stability at the bottom and an etherpad link pointing to a list of use cases for Zaqar. Hi Flavio. This was an interesting read. As somebody whose attention has recently been drawn to Zaqar, I am quite interested in seeing it graduate. # Concerns - Concern on operational burden of requiring NoSQL deploy expertise to the mix of openstack operational skills For those of you not familiar with Zaqar, it currently supports 2 nosql drivers - MongoDB and Redis - and those are the only 2 drivers it supports for now. This will require operators willing to use Zaqar to maintain a new (?) NoSQL technology in their system. Before expressing our thoughts on this matter, let me say that: 1. By removing the SQLAlchemy driver, we basically removed the chance for operators to use an already deployed OpenStack-technology 2. Zaqar won't be backed by any AMQP based messaging technology for now. Here's[0] a summary of the research the team (mostly done by Victoria) did during Juno 3. We (OpenStack) used to require Redis for the zmq matchmaker 4. We (OpenStack) also use memcached for caching and as the oslo caching lib becomes available - or a wrapper on top of dogpile.cache - Redis may be used in place of memcached in more and more deployments. 5. Ceilometer's recommended storage driver is still MongoDB, although Ceilometer has now support for sqlalchemy. (Please correct me if I'm wrong). That being said, it's obvious we already, to some extent, promote some NoSQL technologies. However, for the sake of the discussion, lets assume we don't. I truly believe, with my OpenStack (not Zaqar's) hat on, that we can't keep avoiding these technologies. NoSQL technologies have been around for years and we should be prepared - including OpenStack operators - to support these technologies. Not every tool is good for all tasks - one of the reasons we removed the sqlalchemy driver in the first place - therefore it's impossible to keep an homogeneous environment for all services. I whole heartedly agree that non traditional storage technologies that are becoming mainstream are good candidates for use cases where SQL based storage gets in the way. I wish there wasn't so much FUD (warranted or not) about MongoDB, but that is the reality we live in. With this, I'm not suggesting to ignore the risks and the extra burden this adds but, instead of attempting to avoid it completely by not evolving the stack of services we provide, we should probably work on defining a reasonable subset of NoSQL services we are OK with supporting. This will help making the burden smaller and it'll give operators the option to choose. [0] http://blog.flaper87.com/post/marconi-amqp-see-you-later/ - Concern on should we really reinvent a queue system rather than piggyback on one As mentioned in the meeting on Tuesday, Zaqar is not reinventing message brokers. Zaqar provides a service akin to SQS from AWS with an OpenStack flavor on top. [0] I think Zaqar is more like SMTP and IMAP than AMQP. You're not really trying to connect two processes in real time. You're trying to do fully asynchronous messaging with fully randomized access to any message. Perhaps somebody should explore whether the approaches taken by large scale IMAP providers could be applied to Zaqar. Anyway, I can't imagine writing a system to intentionally use the semantics of IMAP and SMTP. I'd be very interested in seeing actual use cases for it, apologies if those have been posted before. It seems like you're EITHER describing something called XMPP that has at least one open source scalable backend called ejabberd. OR, you've actually hit the nail on the head with bringing up SMTP and IMAP but for some reason that feels strange. SMTP and IMAP already implement every feature you've described, as well as retries/failover/HA and a fully end to end secure transport (if installed properly) If you don't actually set them up to run as a public messaging interface but just as a cloud-local exchange, then you could get by with very low overhead for a massive throughput - it can very easily be run on a single machine for Sean's simplicity, and could just as easily be scaled out using well known techniques for public cloud sized deployments? So why not use existing daemons that do this? You could still use the REST API you've got, but instead of writing it to a mongo backend and trying to implement all of
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
Excerpts from Zane Bitter's message of 2014-09-11 15:21:26 -0700: On 09/09/14 19:56, Clint Byrum wrote: Excerpts from Samuel Merritt's message of 2014-09-09 16:12:09 -0700: On 9/9/14, 12:03 PM, Monty Taylor wrote: On 09/04/2014 01:30 AM, Clint Byrum wrote: Excerpts from Flavio Percoco's message of 2014-09-04 00:08:47 -0700: Greetings, Last Tuesday the TC held the first graduation review for Zaqar. During the meeting some concerns arose. I've listed those concerns below with some comments hoping that it will help starting a discussion before the next meeting. In addition, I've added some comments about the project stability at the bottom and an etherpad link pointing to a list of use cases for Zaqar. Hi Flavio. This was an interesting read. As somebody whose attention has recently been drawn to Zaqar, I am quite interested in seeing it graduate. # Concerns - Concern on operational burden of requiring NoSQL deploy expertise to the mix of openstack operational skills For those of you not familiar with Zaqar, it currently supports 2 nosql drivers - MongoDB and Redis - and those are the only 2 drivers it supports for now. This will require operators willing to use Zaqar to maintain a new (?) NoSQL technology in their system. Before expressing our thoughts on this matter, let me say that: 1. By removing the SQLAlchemy driver, we basically removed the chance for operators to use an already deployed OpenStack-technology 2. Zaqar won't be backed by any AMQP based messaging technology for now. Here's[0] a summary of the research the team (mostly done by Victoria) did during Juno 3. We (OpenStack) used to require Redis for the zmq matchmaker 4. We (OpenStack) also use memcached for caching and as the oslo caching lib becomes available - or a wrapper on top of dogpile.cache - Redis may be used in place of memcached in more and more deployments. 5. Ceilometer's recommended storage driver is still MongoDB, although Ceilometer has now support for sqlalchemy. (Please correct me if I'm wrong). That being said, it's obvious we already, to some extent, promote some NoSQL technologies. However, for the sake of the discussion, lets assume we don't. I truly believe, with my OpenStack (not Zaqar's) hat on, that we can't keep avoiding these technologies. NoSQL technologies have been around for years and we should be prepared - including OpenStack operators - to support these technologies. Not every tool is good for all tasks - one of the reasons we removed the sqlalchemy driver in the first place - therefore it's impossible to keep an homogeneous environment for all services. I whole heartedly agree that non traditional storage technologies that are becoming mainstream are good candidates for use cases where SQL based storage gets in the way. I wish there wasn't so much FUD (warranted or not) about MongoDB, but that is the reality we live in. With this, I'm not suggesting to ignore the risks and the extra burden this adds but, instead of attempting to avoid it completely by not evolving the stack of services we provide, we should probably work on defining a reasonable subset of NoSQL services we are OK with supporting. This will help making the burden smaller and it'll give operators the option to choose. [0] http://blog.flaper87.com/post/marconi-amqp-see-you-later/ - Concern on should we really reinvent a queue system rather than piggyback on one As mentioned in the meeting on Tuesday, Zaqar is not reinventing message brokers. Zaqar provides a service akin to SQS from AWS with an OpenStack flavor on top. [0] I think Zaqar is more like SMTP and IMAP than AMQP. You're not really trying to connect two processes in real time. You're trying to do fully asynchronous messaging with fully randomized access to any message. Perhaps somebody should explore whether the approaches taken by large scale IMAP providers could be applied to Zaqar. Anyway, I can't imagine writing a system to intentionally use the semantics of IMAP and SMTP. I'd be very interested in seeing actual use cases for it, apologies if those have been posted before. It seems like you're EITHER describing something called XMPP that has at least one open source scalable backend called ejabberd. OR, you've actually hit the nail on the head with bringing up SMTP and IMAP but for some reason that feels strange. SMTP and IMAP already implement every feature you've described, as well as retries/failover/HA and a fully end to end secure transport (if installed properly) If you don't actually set them up to run as a public messaging interface but just as a cloud-local exchange, then you could get by with very low overhead for a massive throughput - it can very easily be run on a single machine for Sean's simplicity, and could just as easily be
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
Excerpts from Samuel Merritt's message of 2014-09-09 19:04:58 -0700: On 9/9/14, 4:47 PM, Devananda van der Veen wrote: On Tue, Sep 9, 2014 at 4:12 PM, Samuel Merritt s...@swiftstack.com wrote: On 9/9/14, 12:03 PM, Monty Taylor wrote: [snip] So which is it? Because it sounds like to me it's a thing that actually does NOT need to diverge in technology in any way, but that I've been told that it needs to diverge because it's delivering a different set of features - and I'm pretty sure if it _is_ the thing that needs to diverge in technology because of its feature set, then it's a thing I don't think we should be implementing in python in OpenStack because it already exists and it's called AMQP. Whether Zaqar is more like AMQP or more like email is a really strange metric to use for considering its inclusion. I don't find this strange at all -- I had been judging the technical merits of Zaqar (ex-Marconi) for the last ~18 months based on the understanding that it aimed to provide Queueing-as-a-Service, and found its delivery of that to be lacking on technical grounds. The implementation did not meet my view of what a queue service should provide; it is based on some serious antipatterns (storing a queue in an RDBMS is probably the most obvious); and in fact, it isn't even queue-like in the access patterns enabled by the REST API (random access to a set != a queue). That was the basis for a large part of my objections to the project over time, and a source of frustration for me as the developers justified many of their positions rather than accepted feedback and changed course during the incubation period. The reason for this seems clear now... As was pointed out in the TC meeting today, Zaqar is (was?) actually aiming to provide Messaging-as-a-Service -- not queueing as a service! This is another way of saying it's more like email and less like AMQP, which means my but-its-not-a-queue objection to the project's graduation is irrelevant, and I need to rethink about all my previous assessments of the project. The questions now before us are: - should OpenStack include, in the integrated release, a messaging-as-a-service component? I certainly think so. I've worked on a few reasonable-scale web applications, and they all followed the same pattern: HTTP app servers serving requests quickly, background workers for long-running tasks, and some sort of durable message-broker/queue-server thing for conveying work from the first to the second. A quick straw poll of my nearby coworkers shows that every non-trivial web application that they've worked on in the last decade follows the same pattern. While not *every* application needs such a thing, web apps are quite common these days, and Zaqar satisfies one of their big requirements. Not only that, it does so in a way that requires much less babysitting than run-your-own-broker does. I think you missed the distinction. What you describe is _message queueing_. Not messaging. The difference being the durability and addressability of each message. As Devananda pointed out, a queue doesn't allow addressing the items in the queue directly. You can generally only send, receive, ACK, or NACK. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/10/2014 01:47 AM, Devananda van der Veen wrote: On Tue, Sep 9, 2014 at 4:12 PM, Samuel Merritt s...@swiftstack.com wrote: On 9/9/14, 12:03 PM, Monty Taylor wrote: [snip] So which is it? Because it sounds like to me it's a thing that actually does NOT need to diverge in technology in any way, but that I've been told that it needs to diverge because it's delivering a different set of features - and I'm pretty sure if it _is_ the thing that needs to diverge in technology because of its feature set, then it's a thing I don't think we should be implementing in python in OpenStack because it already exists and it's called AMQP. Whether Zaqar is more like AMQP or more like email is a really strange metric to use for considering its inclusion. I don't find this strange at all -- I had been judging the technical merits of Zaqar (ex-Marconi) for the last ~18 months based on the understanding that it aimed to provide Queueing-as-a-Service, and found its delivery of that to be lacking on technical grounds. The implementation did not meet my view of what a queue service should provide; it is based on some serious antipatterns (storing a queue in an RDBMS is probably the most obvious); and in fact, it isn't even queue-like in the access patterns enabled by the REST API (random access to a set != a queue). That was the basis for a large part of my objections to the project over time, and a source of frustration for me as the developers justified many of their positions rather than accepted feedback and changed course during the incubation period. The reason for this seems clear now... Let me clarify some points, again: 1. We have never said that our recommended driver is an RBDMS. That driver was added for reasons that were already explained in this thread.[0] 2. The get-message-by-id feature/bug was not added as 'queue' feature. When we added it, we were working on messages pagination, that is, a way to iterate and keep getting messages out of the queue by following the lead of the last message. When the pagination thing was completed, adding the get-message-by-id endpoint was easy and cheap, hence we added it. Our current API version is 1.1 and we *can't* remove that endpoint until the next major version. 3. You saying that we just justified our positions rather than accepting the feedback is what frustrates me the most and it just means, with all due respect, that you haven't followed the project close enough. We've addressed feedback from *EVERYONE* whenever that was possible. As far as I can see, your frustration is mostly related to the fact that we haven't removed the `get-message-by-id` endpoint but as I've mentioned, we can't do it right now. However, we have talked about it several times in meetings and a couple of times on the mailing list.[1][2] [0] http://lists.openstack.org/pipermail/openstack-dev/2014-March/030367.html [1] http://lists.openstack.org/pipermail/openstack-dev/2014-May/036131.html [2] http://lists.openstack.org/pipermail/openstack-dev/2014-August/044213.html As was pointed out in the TC meeting today, Zaqar is (was?) actually aiming to provide Messaging-as-a-Service -- not queueing as a service! This is another way of saying it's more like email and less like AMQP, which means my but-its-not-a-queue objection to the project's graduation is irrelevant, and I need to rethink about all my previous assessments of the project. Zaqar is a messaging service, we changed that almost right before the Juno summit to help clarify the project goals. [0] [0] https://wiki.openstack.org/wiki/Zaqar#Overview The questions now before us are: - should OpenStack include, in the integrated release, a messaging-as-a-service component? - is Zaqar a technically sound implementation of such a service? I believe there is. I've been collecting use-cases from integrated projects and other users. I've put them here[0] [0] https://etherpad.openstack.org/p/zaqar-integrated-projects-use-cases As an aside, there are still references to Zaqar as a queue in both the wiki [0], in the governance repo [1], and on launchpad [2]. I'm sorry about this, apparently the change was not spread enough throughout our community, again. Thanks for your feedback, Flavio [0] Multi-tenant queues based on Keystone project IDs https://wiki.openstack.org/wiki/Zaqar#Key_features [1] Queue service is even the official OpenStack Program name, and the mission statement starts with To produce an OpenStack message queueing API and service. http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n315 [2] Zaqar is a new OpenStack project to create a multi-tenant cloud queuing service https://launchpad.net/zaqar ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On Tue, 9 Sep 2014, Samuel Merritt wrote: On 9/9/14, 4:47 PM, Devananda van der Veen wrote: The questions now before us are: - should OpenStack include, in the integrated release, a messaging-as-a-service component? I certainly think so. I've worked on a few reasonable-scale web applications, and they all followed the same pattern: HTTP app servers serving requests quickly, background workers for long-running tasks, and some sort of durable message-broker/queue-server thing for conveying work from the first to the second. A quick straw poll of my nearby coworkers shows that every non-trivial web application that they've worked on in the last decade follows the same pattern. While not *every* application needs such a thing, web apps are quite common these days, and Zaqar satisfies one of their big requirements. Not only that, it does so in a way that requires much less babysitting than run-your-own-broker does. I don't think there's any question about the value of a durable message-broke/queue-server thing in the general case. The question is whether OpenStack is in the business of satisfying that case. Which leads inevitably to the existential questions of What is OpenStack in the business of? and How do we stay sane while being in that business? Every long thread over the last couple of months has trended towards those questions. It's getting pretty tiresome. We'd all be a lot more focused if we knew the answer. -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/10/2014 12:47 AM, Devananda van der Veen wrote: As was pointed out in the TC meeting today, Zaqar is (was?) actually aiming to provide Messaging-as-a-Service -- not queueing as a service! This is another way of saying it's more like email and less like AMQP The glossary[1] describes a model that is much more like messaging-oriented-middleware (AMQP and other things like it) than email. To me, messaging-as-a-service is perhaps a more general purpose term. The concept of a 'queue' might not be exposed in such a system (e.g. MQTT), whereas the term queueing-as-a-service implies that perhaps it is. I don't see them as necessarily distinct things though. The model exposed by Zaqar certainly has 'queues' featured prominently. [1] http://docs.openstack.org/developer/marconi/glossary.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/05/2014 12:36 PM, Tim Bell wrote: How can the average deployer know whether a stackforge is a. An early prototype which has completed (such as some of the early LBaaS packages) b. A project which has lost its initial steam and further investment is not foreseen c. A reliable project where there has not been a significant need to change recently but is a good long term bet This pops up often and to me it looks like a software procurement issue, something that the 'buyer' needs to be able to sort out either with the help of their vendors (distributions or system integrators), or if they go DIY, with other instruments to check code quality, level of support by the community, professional support etc (there are a few EU funded researches and tools, like http://qualipso.icmc.usp.br/OMM/). Why do you think any open source project should be in the business of providing such assurances? Isn't that a role more suitable for the commercial ecosystem? /stef -- Ask and answer questions on https://ask.openstack.org ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
-Original Message- From: Stefano Maffulli [mailto:stef...@openstack.org] Sent: 10 September 2014 19:29 To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting On 09/05/2014 12:36 PM, Tim Bell wrote: How can the average deployer know whether a stackforge is a. An early prototype which has completed (such as some of the early LBaaS packages) b. A project which has lost its initial steam and further investment is not foreseen c. A reliable project where there has not been a significant need to change recently but is a good long term bet This pops up often and to me it looks like a software procurement issue, something that the 'buyer' needs to be able to sort out either with the help of their vendors (distributions or system integrators), or if they go DIY, with other instruments to check code quality, level of support by the community, professional support etc (there are a few EU funded researches and tools, like http://qualipso.icmc.usp.br/OMM/). Why do you think any open source project should be in the business of providing such assurances? Isn't that a role more suitable for the commercial ecosystem? There is an impression that open source software does not have procurement issues. Where we choose to invest manpower in testing, documentation, bug reporting, scaling is as much as commitment as those who choose to buy software. CERN does not have resources on its own to keep a project alive. Combined with others, I'm OK to assist. If there is significant doubt, I will not invest at all, no additional function for my users and the overall community loses. Tim /stef -- Ask and answer questions on https://ask.openstack.org ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/09/2014 07:04 PM, Samuel Merritt wrote: On 9/9/14, 4:47 PM, Devananda van der Veen wrote: On Tue, Sep 9, 2014 at 4:12 PM, Samuel Merritt s...@swiftstack.com wrote: On 9/9/14, 12:03 PM, Monty Taylor wrote: [snip] So which is it? Because it sounds like to me it's a thing that actually does NOT need to diverge in technology in any way, but that I've been told that it needs to diverge because it's delivering a different set of features - and I'm pretty sure if it _is_ the thing that needs to diverge in technology because of its feature set, then it's a thing I don't think we should be implementing in python in OpenStack because it already exists and it's called AMQP. Whether Zaqar is more like AMQP or more like email is a really strange metric to use for considering its inclusion. I don't find this strange at all -- I had been judging the technical merits of Zaqar (ex-Marconi) for the last ~18 months based on the understanding that it aimed to provide Queueing-as-a-Service, and found its delivery of that to be lacking on technical grounds. The implementation did not meet my view of what a queue service should provide; it is based on some serious antipatterns (storing a queue in an RDBMS is probably the most obvious); and in fact, it isn't even queue-like in the access patterns enabled by the REST API (random access to a set != a queue). That was the basis for a large part of my objections to the project over time, and a source of frustration for me as the developers justified many of their positions rather than accepted feedback and changed course during the incubation period. The reason for this seems clear now... As was pointed out in the TC meeting today, Zaqar is (was?) actually aiming to provide Messaging-as-a-Service -- not queueing as a service! This is another way of saying it's more like email and less like AMQP, which means my but-its-not-a-queue objection to the project's graduation is irrelevant, and I need to rethink about all my previous assessments of the project. The questions now before us are: - should OpenStack include, in the integrated release, a messaging-as-a-service component? I certainly think so. I've worked on a few reasonable-scale web applications, and they all followed the same pattern: HTTP app servers serving requests quickly, background workers for long-running tasks, and some sort of durable message-broker/queue-server thing for conveying work from the first to the second. A quick straw poll of my nearby coworkers shows that every non-trivial web application that they've worked on in the last decade follows the same pattern. While not *every* application needs such a thing, web apps are quite common these days, and Zaqar satisfies one of their big requirements. Not only that, it does so in a way that requires much less babysitting than run-your-own-broker does. Right. But here's the thing. What you just described is what we all thought zaqar was aiming to be in the beginning. We did not think it was a GOOD implementation of that, so while we agreed that it would be useful to have one of those, we were not crazy about the implementation. But that is what we have recently learned that zaqar is NOT. It is not a message queue in the way that you describe. It is something else. Now, I tend to believe that that something else is also potentially interesting - but the question of whether that thing is a thing that we want, and whether _this_ implementation of that this is a good implementation are the questions at hand. - is Zaqar a technically sound implementation of such a service? As an aside, there are still references to Zaqar as a queue in both the wiki [0], in the governance repo [1], and on launchpad [2]. Regards, Devananda [0] Multi-tenant queues based on Keystone project IDs https://wiki.openstack.org/wiki/Zaqar#Key_features [1] Queue service is even the official OpenStack Program name, and the mission statement starts with To produce an OpenStack message queueing API and service. http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n315 [2] Zaqar is a new OpenStack project to create a multi-tenant cloud queuing service https://launchpad.net/zaqar ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/10/2014 10:29 AM, Stefano Maffulli wrote: On 09/05/2014 12:36 PM, Tim Bell wrote: How can the average deployer know whether a stackforge is a. An early prototype which has completed (such as some of the early LBaaS packages) b. A project which has lost its initial steam and further investment is not foreseen c. A reliable project where there has not been a significant need to change recently but is a good long term bet This pops up often and to me it looks like a software procurement issue, something that the 'buyer' needs to be able to sort out either with the help of their vendors (distributions or system integrators), or if they go DIY, with other instruments to check code quality, level of support by the community, professional support etc (there are a few EU funded researches and tools, like http://qualipso.icmc.usp.br/OMM/). Why do you think any open source project should be in the business of providing such assurances? Isn't that a role more suitable for the commercial ecosystem? I reject soundly and fundamentally the idea that Open Source projects NEED a commercial ecosystem to provide solid quality software. That is a description of a thing called Open Core and every time it has come up in the context of OpenStack I and others have been quite successful in reiterating: OpenStack is NOT Open Core bind has been running DNS for the world for 20 years and does it quite admirably without someone needing to buy Commercial Bind Apache has been the dominant web server for forever, and if there is any human who has ever purchased Commerical Apache they are an absolute moron. There is no such thing as Commercial Python Open Source software has been running the entire internet, at scale, since before the Internet was wide spread, and since well before the term Open Source existed. In fact, Open Source software drove the world of computing before the idea of Commercial Software was invented. Does this mean that our commercial vendors are bad? No. Absolutely not. I love each and every one of them and want all of them to make tons of money. But if the thing I produce isn't good in the first place, I expect them to show up and help me fix it. Here. In the Open Source project. Otherwise, they are a leech and should be mocked. So, no, I do not believe that it should be the purview of the commercial ecosystem to provide quality assurances about the Open Source software that I work on. And I'm highly offended that as a community manager for an Open Source project, that you seem to think that the output of our project should not be world class all by itself. Monty ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/10/2014 12:56 PM, Monty Taylor wrote: I reject soundly and fundamentally the idea that Open Source projects NEED a commercial ecosystem to provide solid quality software. That's not what I said. I said that assuring the quality of code on a public repository is not necessarily something that can be asked to the maintainers of the code. A very simple example: a repository where nobody committed anything for a long time. If nobody seems to be maintaining the code, how can a user get an assurance of the quality of its code? That's all I meant to say: quality of open source code (just like non-free software code) is not necessarily something that can be left to the maintainers of the code. It's like asking the waiter if the wine is good... of course the answer is yes. That is a description of a thing called Open Core So not. Open Core is a different thing and is orthogonal to code quality, quality and width of support, maintenance etc. Apache has been the dominant web server for forever, and if there is any human who has ever purchased Commerical Apache they are an absolute moron. And yet, Red Hat sells quite a lot of software that is completely free as in beer (in addition to being free as in freedom). This is what I'm talking about, not 'open core'. To go back in topic, I was explicitly addressing Tim's question How can the average deployer know whether a stackforge is [...] and my answer is that stackforge can have perfectly fine code while there could be bad code in openstack/*. And I simply don't think that the openstack project as a whole can answer anything but all our code is great, if asked. Therefore, deciding whether code on stackforge or anywhere else for that matter, needs to be properly evaluated at procurement time. On 09/10/2014 10:45 AM, Tim Bell wrote: There is an impression that open source software does not have procurement issues. I guarantee you I never had that impression, quite the contrary instead. I've worked with Italian government and large corporations to teach them how to deal with open source code long since there was no Red Hat in Italy. Also, to clarify my answer to Tim, I reject the idea that code in stackforge is a second class citizen. As simple as that. /stef -- Ask and answer questions on https://ask.openstack.org ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/04/2014 01:30 AM, Clint Byrum wrote: Excerpts from Flavio Percoco's message of 2014-09-04 00:08:47 -0700: Greetings, Last Tuesday the TC held the first graduation review for Zaqar. During the meeting some concerns arose. I've listed those concerns below with some comments hoping that it will help starting a discussion before the next meeting. In addition, I've added some comments about the project stability at the bottom and an etherpad link pointing to a list of use cases for Zaqar. Hi Flavio. This was an interesting read. As somebody whose attention has recently been drawn to Zaqar, I am quite interested in seeing it graduate. # Concerns - Concern on operational burden of requiring NoSQL deploy expertise to the mix of openstack operational skills For those of you not familiar with Zaqar, it currently supports 2 nosql drivers - MongoDB and Redis - and those are the only 2 drivers it supports for now. This will require operators willing to use Zaqar to maintain a new (?) NoSQL technology in their system. Before expressing our thoughts on this matter, let me say that: 1. By removing the SQLAlchemy driver, we basically removed the chance for operators to use an already deployed OpenStack-technology 2. Zaqar won't be backed by any AMQP based messaging technology for now. Here's[0] a summary of the research the team (mostly done by Victoria) did during Juno 3. We (OpenStack) used to require Redis for the zmq matchmaker 4. We (OpenStack) also use memcached for caching and as the oslo caching lib becomes available - or a wrapper on top of dogpile.cache - Redis may be used in place of memcached in more and more deployments. 5. Ceilometer's recommended storage driver is still MongoDB, although Ceilometer has now support for sqlalchemy. (Please correct me if I'm wrong). That being said, it's obvious we already, to some extent, promote some NoSQL technologies. However, for the sake of the discussion, lets assume we don't. I truly believe, with my OpenStack (not Zaqar's) hat on, that we can't keep avoiding these technologies. NoSQL technologies have been around for years and we should be prepared - including OpenStack operators - to support these technologies. Not every tool is good for all tasks - one of the reasons we removed the sqlalchemy driver in the first place - therefore it's impossible to keep an homogeneous environment for all services. I whole heartedly agree that non traditional storage technologies that are becoming mainstream are good candidates for use cases where SQL based storage gets in the way. I wish there wasn't so much FUD (warranted or not) about MongoDB, but that is the reality we live in. With this, I'm not suggesting to ignore the risks and the extra burden this adds but, instead of attempting to avoid it completely by not evolving the stack of services we provide, we should probably work on defining a reasonable subset of NoSQL services we are OK with supporting. This will help making the burden smaller and it'll give operators the option to choose. [0] http://blog.flaper87.com/post/marconi-amqp-see-you-later/ - Concern on should we really reinvent a queue system rather than piggyback on one As mentioned in the meeting on Tuesday, Zaqar is not reinventing message brokers. Zaqar provides a service akin to SQS from AWS with an OpenStack flavor on top. [0] I think Zaqar is more like SMTP and IMAP than AMQP. You're not really trying to connect two processes in real time. You're trying to do fully asynchronous messaging with fully randomized access to any message. Perhaps somebody should explore whether the approaches taken by large scale IMAP providers could be applied to Zaqar. Anyway, I can't imagine writing a system to intentionally use the semantics of IMAP and SMTP. I'd be very interested in seeing actual use cases for it, apologies if those have been posted before. It seems like you're EITHER describing something called XMPP that has at least one open source scalable backend called ejabberd. OR, you've actually hit the nail on the head with bringing up SMTP and IMAP but for some reason that feels strange. SMTP and IMAP already implement every feature you've described, as well as retries/failover/HA and a fully end to end secure transport (if installed properly) If you don't actually set them up to run as a public messaging interface but just as a cloud-local exchange, then you could get by with very low overhead for a massive throughput - it can very easily be run on a single machine for Sean's simplicity, and could just as easily be scaled out using well known techniques for public cloud sized deployments? So why not use existing daemons that do this? You could still use the REST API you've got, but instead of writing it to a mongo backend and trying to implement all of the things that already exist in SMTP/IMAP - you could just have them front to it. You could even bypass normal delivery mechanisms and do
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 9/9/14, 12:03 PM, Monty Taylor wrote: On 09/04/2014 01:30 AM, Clint Byrum wrote: Excerpts from Flavio Percoco's message of 2014-09-04 00:08:47 -0700: Greetings, Last Tuesday the TC held the first graduation review for Zaqar. During the meeting some concerns arose. I've listed those concerns below with some comments hoping that it will help starting a discussion before the next meeting. In addition, I've added some comments about the project stability at the bottom and an etherpad link pointing to a list of use cases for Zaqar. Hi Flavio. This was an interesting read. As somebody whose attention has recently been drawn to Zaqar, I am quite interested in seeing it graduate. # Concerns - Concern on operational burden of requiring NoSQL deploy expertise to the mix of openstack operational skills For those of you not familiar with Zaqar, it currently supports 2 nosql drivers - MongoDB and Redis - and those are the only 2 drivers it supports for now. This will require operators willing to use Zaqar to maintain a new (?) NoSQL technology in their system. Before expressing our thoughts on this matter, let me say that: 1. By removing the SQLAlchemy driver, we basically removed the chance for operators to use an already deployed OpenStack-technology 2. Zaqar won't be backed by any AMQP based messaging technology for now. Here's[0] a summary of the research the team (mostly done by Victoria) did during Juno 3. We (OpenStack) used to require Redis for the zmq matchmaker 4. We (OpenStack) also use memcached for caching and as the oslo caching lib becomes available - or a wrapper on top of dogpile.cache - Redis may be used in place of memcached in more and more deployments. 5. Ceilometer's recommended storage driver is still MongoDB, although Ceilometer has now support for sqlalchemy. (Please correct me if I'm wrong). That being said, it's obvious we already, to some extent, promote some NoSQL technologies. However, for the sake of the discussion, lets assume we don't. I truly believe, with my OpenStack (not Zaqar's) hat on, that we can't keep avoiding these technologies. NoSQL technologies have been around for years and we should be prepared - including OpenStack operators - to support these technologies. Not every tool is good for all tasks - one of the reasons we removed the sqlalchemy driver in the first place - therefore it's impossible to keep an homogeneous environment for all services. I whole heartedly agree that non traditional storage technologies that are becoming mainstream are good candidates for use cases where SQL based storage gets in the way. I wish there wasn't so much FUD (warranted or not) about MongoDB, but that is the reality we live in. With this, I'm not suggesting to ignore the risks and the extra burden this adds but, instead of attempting to avoid it completely by not evolving the stack of services we provide, we should probably work on defining a reasonable subset of NoSQL services we are OK with supporting. This will help making the burden smaller and it'll give operators the option to choose. [0] http://blog.flaper87.com/post/marconi-amqp-see-you-later/ - Concern on should we really reinvent a queue system rather than piggyback on one As mentioned in the meeting on Tuesday, Zaqar is not reinventing message brokers. Zaqar provides a service akin to SQS from AWS with an OpenStack flavor on top. [0] I think Zaqar is more like SMTP and IMAP than AMQP. You're not really trying to connect two processes in real time. You're trying to do fully asynchronous messaging with fully randomized access to any message. Perhaps somebody should explore whether the approaches taken by large scale IMAP providers could be applied to Zaqar. Anyway, I can't imagine writing a system to intentionally use the semantics of IMAP and SMTP. I'd be very interested in seeing actual use cases for it, apologies if those have been posted before. It seems like you're EITHER describing something called XMPP that has at least one open source scalable backend called ejabberd. OR, you've actually hit the nail on the head with bringing up SMTP and IMAP but for some reason that feels strange. SMTP and IMAP already implement every feature you've described, as well as retries/failover/HA and a fully end to end secure transport (if installed properly) If you don't actually set them up to run as a public messaging interface but just as a cloud-local exchange, then you could get by with very low overhead for a massive throughput - it can very easily be run on a single machine for Sean's simplicity, and could just as easily be scaled out using well known techniques for public cloud sized deployments? So why not use existing daemons that do this? You could still use the REST API you've got, but instead of writing it to a mongo backend and trying to implement all of the things that already exist in SMTP/IMAP - you could just have them front to it. You could even bypass normal
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On Tue, Sep 9, 2014 at 4:12 PM, Samuel Merritt s...@swiftstack.com wrote: On 9/9/14, 12:03 PM, Monty Taylor wrote: [snip] So which is it? Because it sounds like to me it's a thing that actually does NOT need to diverge in technology in any way, but that I've been told that it needs to diverge because it's delivering a different set of features - and I'm pretty sure if it _is_ the thing that needs to diverge in technology because of its feature set, then it's a thing I don't think we should be implementing in python in OpenStack because it already exists and it's called AMQP. Whether Zaqar is more like AMQP or more like email is a really strange metric to use for considering its inclusion. I don't find this strange at all -- I had been judging the technical merits of Zaqar (ex-Marconi) for the last ~18 months based on the understanding that it aimed to provide Queueing-as-a-Service, and found its delivery of that to be lacking on technical grounds. The implementation did not meet my view of what a queue service should provide; it is based on some serious antipatterns (storing a queue in an RDBMS is probably the most obvious); and in fact, it isn't even queue-like in the access patterns enabled by the REST API (random access to a set != a queue). That was the basis for a large part of my objections to the project over time, and a source of frustration for me as the developers justified many of their positions rather than accepted feedback and changed course during the incubation period. The reason for this seems clear now... As was pointed out in the TC meeting today, Zaqar is (was?) actually aiming to provide Messaging-as-a-Service -- not queueing as a service! This is another way of saying it's more like email and less like AMQP, which means my but-its-not-a-queue objection to the project's graduation is irrelevant, and I need to rethink about all my previous assessments of the project. The questions now before us are: - should OpenStack include, in the integrated release, a messaging-as-a-service component? - is Zaqar a technically sound implementation of such a service? As an aside, there are still references to Zaqar as a queue in both the wiki [0], in the governance repo [1], and on launchpad [2]. Regards, Devananda [0] Multi-tenant queues based on Keystone project IDs https://wiki.openstack.org/wiki/Zaqar#Key_features [1] Queue service is even the official OpenStack Program name, and the mission statement starts with To produce an OpenStack message queueing API and service. http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n315 [2] Zaqar is a new OpenStack project to create a multi-tenant cloud queuing service https://launchpad.net/zaqar ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
Excerpts from Samuel Merritt's message of 2014-09-09 16:12:09 -0700: On 9/9/14, 12:03 PM, Monty Taylor wrote: On 09/04/2014 01:30 AM, Clint Byrum wrote: Excerpts from Flavio Percoco's message of 2014-09-04 00:08:47 -0700: Greetings, Last Tuesday the TC held the first graduation review for Zaqar. During the meeting some concerns arose. I've listed those concerns below with some comments hoping that it will help starting a discussion before the next meeting. In addition, I've added some comments about the project stability at the bottom and an etherpad link pointing to a list of use cases for Zaqar. Hi Flavio. This was an interesting read. As somebody whose attention has recently been drawn to Zaqar, I am quite interested in seeing it graduate. # Concerns - Concern on operational burden of requiring NoSQL deploy expertise to the mix of openstack operational skills For those of you not familiar with Zaqar, it currently supports 2 nosql drivers - MongoDB and Redis - and those are the only 2 drivers it supports for now. This will require operators willing to use Zaqar to maintain a new (?) NoSQL technology in their system. Before expressing our thoughts on this matter, let me say that: 1. By removing the SQLAlchemy driver, we basically removed the chance for operators to use an already deployed OpenStack-technology 2. Zaqar won't be backed by any AMQP based messaging technology for now. Here's[0] a summary of the research the team (mostly done by Victoria) did during Juno 3. We (OpenStack) used to require Redis for the zmq matchmaker 4. We (OpenStack) also use memcached for caching and as the oslo caching lib becomes available - or a wrapper on top of dogpile.cache - Redis may be used in place of memcached in more and more deployments. 5. Ceilometer's recommended storage driver is still MongoDB, although Ceilometer has now support for sqlalchemy. (Please correct me if I'm wrong). That being said, it's obvious we already, to some extent, promote some NoSQL technologies. However, for the sake of the discussion, lets assume we don't. I truly believe, with my OpenStack (not Zaqar's) hat on, that we can't keep avoiding these technologies. NoSQL technologies have been around for years and we should be prepared - including OpenStack operators - to support these technologies. Not every tool is good for all tasks - one of the reasons we removed the sqlalchemy driver in the first place - therefore it's impossible to keep an homogeneous environment for all services. I whole heartedly agree that non traditional storage technologies that are becoming mainstream are good candidates for use cases where SQL based storage gets in the way. I wish there wasn't so much FUD (warranted or not) about MongoDB, but that is the reality we live in. With this, I'm not suggesting to ignore the risks and the extra burden this adds but, instead of attempting to avoid it completely by not evolving the stack of services we provide, we should probably work on defining a reasonable subset of NoSQL services we are OK with supporting. This will help making the burden smaller and it'll give operators the option to choose. [0] http://blog.flaper87.com/post/marconi-amqp-see-you-later/ - Concern on should we really reinvent a queue system rather than piggyback on one As mentioned in the meeting on Tuesday, Zaqar is not reinventing message brokers. Zaqar provides a service akin to SQS from AWS with an OpenStack flavor on top. [0] I think Zaqar is more like SMTP and IMAP than AMQP. You're not really trying to connect two processes in real time. You're trying to do fully asynchronous messaging with fully randomized access to any message. Perhaps somebody should explore whether the approaches taken by large scale IMAP providers could be applied to Zaqar. Anyway, I can't imagine writing a system to intentionally use the semantics of IMAP and SMTP. I'd be very interested in seeing actual use cases for it, apologies if those have been posted before. It seems like you're EITHER describing something called XMPP that has at least one open source scalable backend called ejabberd. OR, you've actually hit the nail on the head with bringing up SMTP and IMAP but for some reason that feels strange. SMTP and IMAP already implement every feature you've described, as well as retries/failover/HA and a fully end to end secure transport (if installed properly) If you don't actually set them up to run as a public messaging interface but just as a cloud-local exchange, then you could get by with very low overhead for a massive throughput - it can very easily be run on a single machine for Sean's simplicity, and could just as easily be scaled out using well known techniques for public cloud sized deployments? So why not use existing daemons that
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On Thu, Sep 4, 2014 at 1:44 PM, Kurt Griffiths kurt.griffi...@rackspace.com wrote: [snip] Does a Qpid/Rabbit/Kafka provisioning service make sense? Probably. Would such a service totally overlap in terms of use-cases with Zaqar? Community feedback suggests otherwise. Will there be some other kind of thing that comes out of the woodwork? Possibly. (Heck, if something better comes along I for one have no qualms in shifting resources to the more elegant solution--again, use the best tool for the job.) This process happens all the time in the broader open-source world. But this process takes a healthy amount of time, plus broad exposure and usage, which is something that you simply don’t get as a non-integrated project in the OpenStack ecosystem. While that is de rigueur today, it's actually at the core of the current problem space. Blessing a project by integrating it is not a scalable long-term solution. We don't have a model to integrate 1 project for the same space // of the same type, or to bless the stability of a non-integrated project. You won't see two messaging services, or two compute services, in the integrated release. In fact, integration is supposed to occur only *after* the community has sorted out a winner within a given space. In my view, it should also happen only after the community has proven a project to be stable and scalable in production. It should be self-evident that, for a large and healthy ecosystem of production-quality projects to be created and flourish, we can not pick a winner and shut down competition by integrating a project *prior* to that project getting broad exposure and usage. A practice of integrating projects merely to get them exposure and contributors is self-defeating. In any case, it’s pretty clear to me that Zaqar graduating should not be viewed as making it the officially blessed messaging service for the cloud” That's exactly what graduation does, though. Your statement in the previous paragraph - that non-integrated projects don't get adoption - only furthers this point. and nobody is allowed to have any other ideas, ever. Of course other people can have other ideas -- but we don't have a precedent for handling it inside the community. Look at Ceilometer - there are at least two other projects which attempted to fill that space, but we haven't any means to accept them into OpenStack without either removing Ceilometer or encouraging those projects to merge into Ceilometer. If that happens, it’s only a symptom of a deeper perception/process problem that is far from unique to Zaqar. In fact, I think it touches on all non-integrated projects, and many integrated ones as well. Yup. I agree that we shouldn't hold Zaqar hostage while the community sorts out the small-tent-big-camp questions. But I also feel like we _must_ sort that out soon, because the current system (integrate all the things!) doesn't appear to be sustainable for much longer. -Devananda ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
Excerpts from Devananda van der Veen's message of 2014-09-09 16:47:27 -0700: On Tue, Sep 9, 2014 at 4:12 PM, Samuel Merritt s...@swiftstack.com wrote: On 9/9/14, 12:03 PM, Monty Taylor wrote: [snip] So which is it? Because it sounds like to me it's a thing that actually does NOT need to diverge in technology in any way, but that I've been told that it needs to diverge because it's delivering a different set of features - and I'm pretty sure if it _is_ the thing that needs to diverge in technology because of its feature set, then it's a thing I don't think we should be implementing in python in OpenStack because it already exists and it's called AMQP. Whether Zaqar is more like AMQP or more like email is a really strange metric to use for considering its inclusion. I don't find this strange at all -- I had been judging the technical merits of Zaqar (ex-Marconi) for the last ~18 months based on the understanding that it aimed to provide Queueing-as-a-Service, and found its delivery of that to be lacking on technical grounds. The implementation did not meet my view of what a queue service should provide; it is based on some serious antipatterns (storing a queue in an RDBMS is probably the most obvious); and in fact, it isn't even queue-like in the access patterns enabled by the REST API (random access to a set != a queue). That was the basis for a large part of my objections to the project over time, and a source of frustration for me as the developers justified many of their positions rather than accepted feedback and changed course during the incubation period. The reason for this seems clear now... As was pointed out in the TC meeting today, Zaqar is (was?) actually aiming to provide Messaging-as-a-Service -- not queueing as a service! This is another way of saying it's more like email and less like AMQP, which means my but-its-not-a-queue objection to the project's graduation is irrelevant, and I need to rethink about all my previous assessments of the project. Well said. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
*should OpenStack include, in the integrated release, a messaging-as-a-service component* Assuming this is truly a question that represents where we are and not exploratory of what we might want to address, I would say the answer is a resounding no, as queuing is within the scope of what Openstack is and has always been. If we get into integrated messaging, I'm struggling to understand what value it adds to the IaaS goal. We might as well start integrating office and productivity applications while we're at it. Sorry if i sound cheeky but considering this seems rather odd to me. *Adam Lawson* *CEO, Principal Architect* AQORN, Inc. 427 North Tatnall Street Ste. 58461 Wilmington, Delaware 19801-2230 Toll-free: (844) 4-AQORN-NOW ext. 101 International: +1 302-387-4660 Direct: +1 916-246-2072 On Tue, Sep 9, 2014 at 5:03 PM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Devananda van der Veen's message of 2014-09-09 16:47:27 -0700: On Tue, Sep 9, 2014 at 4:12 PM, Samuel Merritt s...@swiftstack.com wrote: On 9/9/14, 12:03 PM, Monty Taylor wrote: [snip] So which is it? Because it sounds like to me it's a thing that actually does NOT need to diverge in technology in any way, but that I've been told that it needs to diverge because it's delivering a different set of features - and I'm pretty sure if it _is_ the thing that needs to diverge in technology because of its feature set, then it's a thing I don't think we should be implementing in python in OpenStack because it already exists and it's called AMQP. Whether Zaqar is more like AMQP or more like email is a really strange metric to use for considering its inclusion. I don't find this strange at all -- I had been judging the technical merits of Zaqar (ex-Marconi) for the last ~18 months based on the understanding that it aimed to provide Queueing-as-a-Service, and found its delivery of that to be lacking on technical grounds. The implementation did not meet my view of what a queue service should provide; it is based on some serious antipatterns (storing a queue in an RDBMS is probably the most obvious); and in fact, it isn't even queue-like in the access patterns enabled by the REST API (random access to a set != a queue). That was the basis for a large part of my objections to the project over time, and a source of frustration for me as the developers justified many of their positions rather than accepted feedback and changed course during the incubation period. The reason for this seems clear now... As was pointed out in the TC meeting today, Zaqar is (was?) actually aiming to provide Messaging-as-a-Service -- not queueing as a service! This is another way of saying it's more like email and less like AMQP, which means my but-its-not-a-queue objection to the project's graduation is irrelevant, and I need to rethink about all my previous assessments of the project. Well said. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
Devananda, While that is de rigueur today, it's actually at the core of the current problem space. Blessing a project by integrating it is not a scalable long-term solution. We don't have a model to integrate 1 project for the same space // of the same type, or to bless the stability of a non-integrated project. You won't see two messaging services, or two compute services, in the integrated release. In fact, integration is supposed to occur only *after* the community has sorted out a winner within a given space. In my view, it should also happen only after the community has proven a project to be stable and scalable in production. After looking at such profiles: http://boris-42.github.io/ngk.html And getting 150 DB requests (without neutron) to create one single VM, I don't believe that set of current integrated OpenStack projects is scalable well. (I mean without customization) So I would like to say 2 things: - Rules should be the same for all projects (including incubated/integrated) - Nothing should be incubated/integrated. Cause projects have to evolve, to evolve they need competition. In other words, monopoly sux in any moment of time (even after community decided to chose project A and not project B) Best regards, Boris Pavlovic On Wed, Sep 10, 2014 at 4:18 AM, Adam Lawson alaw...@aqorn.com wrote: *should OpenStack include, in the integrated release, a messaging-as-a-service component* Assuming this is truly a question that represents where we are and not exploratory of what we might want to address, I would say the answer is a resounding no, as queuing is within the scope of what Openstack is and has always been. If we get into integrated messaging, I'm struggling to understand what value it adds to the IaaS goal. We might as well start integrating office and productivity applications while we're at it. Sorry if i sound cheeky but considering this seems rather odd to me. *Adam Lawson* *CEO, Principal Architect* AQORN, Inc. 427 North Tatnall Street Ste. 58461 Wilmington, Delaware 19801-2230 Toll-free: (844) 4-AQORN-NOW ext. 101 International: +1 302-387-4660 Direct: +1 916-246-2072 On Tue, Sep 9, 2014 at 5:03 PM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Devananda van der Veen's message of 2014-09-09 16:47:27 -0700: On Tue, Sep 9, 2014 at 4:12 PM, Samuel Merritt s...@swiftstack.com wrote: On 9/9/14, 12:03 PM, Monty Taylor wrote: [snip] So which is it? Because it sounds like to me it's a thing that actually does NOT need to diverge in technology in any way, but that I've been told that it needs to diverge because it's delivering a different set of features - and I'm pretty sure if it _is_ the thing that needs to diverge in technology because of its feature set, then it's a thing I don't think we should be implementing in python in OpenStack because it already exists and it's called AMQP. Whether Zaqar is more like AMQP or more like email is a really strange metric to use for considering its inclusion. I don't find this strange at all -- I had been judging the technical merits of Zaqar (ex-Marconi) for the last ~18 months based on the understanding that it aimed to provide Queueing-as-a-Service, and found its delivery of that to be lacking on technical grounds. The implementation did not meet my view of what a queue service should provide; it is based on some serious antipatterns (storing a queue in an RDBMS is probably the most obvious); and in fact, it isn't even queue-like in the access patterns enabled by the REST API (random access to a set != a queue). That was the basis for a large part of my objections to the project over time, and a source of frustration for me as the developers justified many of their positions rather than accepted feedback and changed course during the incubation period. The reason for this seems clear now... As was pointed out in the TC meeting today, Zaqar is (was?) actually aiming to provide Messaging-as-a-Service -- not queueing as a service! This is another way of saying it's more like email and less like AMQP, which means my but-its-not-a-queue objection to the project's graduation is irrelevant, and I need to rethink about all my previous assessments of the project. Well said. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On Tue, Sep 9, 2014 at 5:31 PM, Boris Pavlovic bo...@pavlovic.me wrote: Devananda, While that is de rigueur today, it's actually at the core of the current problem space. Blessing a project by integrating it is not a scalable long-term solution. We don't have a model to integrate 1 project for the same space // of the same type, or to bless the stability of a non-integrated project. You won't see two messaging services, or two compute services, in the integrated release. In fact, integration is supposed to occur only *after* the community has sorted out a winner within a given space. In my view, it should also happen only after the community has proven a project to be stable and scalable in production. After looking at such profiles: http://boris-42.github.io/ngk.html And getting 150 DB requests (without neutron) to create one single VM, I don't believe that set of current integrated OpenStack projects is scalable well. (I mean without customization) I'm not going to defend the DB performance of Nova or other services. This thread isn't the place for that discussion. So I would like to say 2 things: - Rules should be the same for all projects (including incubated/integrated) Yup. This is why the TC revisits integrated projects once per cycle now, too. - Nothing should be incubated/integrated. This is a blatant straw-man. If you're suggesting we stop all integration testing, release management, etc -- the very things which the integrated release process coordinates... well, I don't think that's what you're saying. Except it is. Cause projects have to evolve, to evolve they need competition. In other words, monopoly sux in any moment of time (even after community decided to chose project A and not project B) In order for a project to evolve, a project needs people contributing to it. More often than not, that is because someone is using the project, and it doesn't do what they want, so they improve it in some way. Incubation was intended to be a signal to early adopters to begin using (and thus, hopefully, contributing to) a project, encouraging collaboration and reducing NIH friction between corporations within the ecosystem. It hasn't gone exactly as planned, but it's also worked fairly well for _this_ purpose, in my opinion. However, adding more and more projects into the integrated release, and thus increasing the testing complexity and imposing greater requirements on operators -- this is an imminent scaling problem, as Sean has eloquently pointed out before in several long email threads which I won't recount here. All of this is to say that Kurt's statement: [You don't get] broad exposure and usage... as a non-integrated project in the OpenStack ecosystem. is an accurate representation of one problem facing OpenStack today. I don't think we solve that problem by following the established norm - we solve it by creating a mechanism for non-integrated projects to get the exposure and usage they need _without_ becoming a burden on our QA, docs, and release teams, and without forcing that project upon operators. But as I said earlier, we shouldn't hold Zaqar hostage while we sort out what that solution looks like... Anyhow, my apologies for the bike shed. I felt it was worth voicing my disagreement with Kurt's statement that graduation should not be viewed as an official blessing of Zaqar as OpenStack's Messaging Service. Today, I believe that's exactly what it is. With that blessing comes an additional burden on the community to support it. Perhaps that will change in the future. -Devananda ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 9/9/14, 4:47 PM, Devananda van der Veen wrote: On Tue, Sep 9, 2014 at 4:12 PM, Samuel Merritt s...@swiftstack.com wrote: On 9/9/14, 12:03 PM, Monty Taylor wrote: [snip] So which is it? Because it sounds like to me it's a thing that actually does NOT need to diverge in technology in any way, but that I've been told that it needs to diverge because it's delivering a different set of features - and I'm pretty sure if it _is_ the thing that needs to diverge in technology because of its feature set, then it's a thing I don't think we should be implementing in python in OpenStack because it already exists and it's called AMQP. Whether Zaqar is more like AMQP or more like email is a really strange metric to use for considering its inclusion. I don't find this strange at all -- I had been judging the technical merits of Zaqar (ex-Marconi) for the last ~18 months based on the understanding that it aimed to provide Queueing-as-a-Service, and found its delivery of that to be lacking on technical grounds. The implementation did not meet my view of what a queue service should provide; it is based on some serious antipatterns (storing a queue in an RDBMS is probably the most obvious); and in fact, it isn't even queue-like in the access patterns enabled by the REST API (random access to a set != a queue). That was the basis for a large part of my objections to the project over time, and a source of frustration for me as the developers justified many of their positions rather than accepted feedback and changed course during the incubation period. The reason for this seems clear now... As was pointed out in the TC meeting today, Zaqar is (was?) actually aiming to provide Messaging-as-a-Service -- not queueing as a service! This is another way of saying it's more like email and less like AMQP, which means my but-its-not-a-queue objection to the project's graduation is irrelevant, and I need to rethink about all my previous assessments of the project. The questions now before us are: - should OpenStack include, in the integrated release, a messaging-as-a-service component? I certainly think so. I've worked on a few reasonable-scale web applications, and they all followed the same pattern: HTTP app servers serving requests quickly, background workers for long-running tasks, and some sort of durable message-broker/queue-server thing for conveying work from the first to the second. A quick straw poll of my nearby coworkers shows that every non-trivial web application that they've worked on in the last decade follows the same pattern. While not *every* application needs such a thing, web apps are quite common these days, and Zaqar satisfies one of their big requirements. Not only that, it does so in a way that requires much less babysitting than run-your-own-broker does. - is Zaqar a technically sound implementation of such a service? As an aside, there are still references to Zaqar as a queue in both the wiki [0], in the governance repo [1], and on launchpad [2]. Regards, Devananda [0] Multi-tenant queues based on Keystone project IDs https://wiki.openstack.org/wiki/Zaqar#Key_features [1] Queue service is even the official OpenStack Program name, and the mission statement starts with To produce an OpenStack message queueing API and service. http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n315 [2] Zaqar is a new OpenStack project to create a multi-tenant cloud queuing service https://launchpad.net/zaqar ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/04/2014 07:08 PM, Clint Byrum wrote: Excerpts from Flavio Percoco's message of 2014-09-04 06:01:45 -0700: On 09/04/2014 02:14 PM, Sean Dague wrote: On 09/04/2014 03:08 AM, Flavio Percoco wrote: Greetings, Last Tuesday the TC held the first graduation review for Zaqar. During the meeting some concerns arose. I've listed those concerns below with some comments hoping that it will help starting a discussion before the next meeting. In addition, I've added some comments about the project stability at the bottom and an etherpad link pointing to a list of use cases for Zaqar. # Concerns - Concern on operational burden of requiring NoSQL deploy expertise to the mix of openstack operational skills For those of you not familiar with Zaqar, it currently supports 2 nosql drivers - MongoDB and Redis - and those are the only 2 drivers it supports for now. This will require operators willing to use Zaqar to maintain a new (?) NoSQL technology in their system. Before expressing our thoughts on this matter, let me say that: 1. By removing the SQLAlchemy driver, we basically removed the chance for operators to use an already deployed OpenStack-technology 2. Zaqar won't be backed by any AMQP based messaging technology for now. Here's[0] a summary of the research the team (mostly done by Victoria) did during Juno 3. We (OpenStack) used to require Redis for the zmq matchmaker 4. We (OpenStack) also use memcached for caching and as the oslo caching lib becomes available - or a wrapper on top of dogpile.cache - Redis may be used in place of memcached in more and more deployments. 5. Ceilometer's recommended storage driver is still MongoDB, although Ceilometer has now support for sqlalchemy. (Please correct me if I'm wrong). That being said, it's obvious we already, to some extent, promote some NoSQL technologies. However, for the sake of the discussion, lets assume we don't. I truly believe, with my OpenStack (not Zaqar's) hat on, that we can't keep avoiding these technologies. NoSQL technologies have been around for years and we should be prepared - including OpenStack operators - to support these technologies. Not every tool is good for all tasks - one of the reasons we removed the sqlalchemy driver in the first place - therefore it's impossible to keep an homogeneous environment for all services. With this, I'm not suggesting to ignore the risks and the extra burden this adds but, instead of attempting to avoid it completely by not evolving the stack of services we provide, we should probably work on defining a reasonable subset of NoSQL services we are OK with supporting. This will help making the burden smaller and it'll give operators the option to choose. [0] http://blog.flaper87.com/post/marconi-amqp-see-you-later/ I've been one of the consistent voices concerned about a hard requirement on adding NoSQL into the mix. So I'll explain that thinking a bit more. I feel like when the TC makes an integration decision previously this has been about evaluating the project applying for integration, and if they met some specific criteria they were told about some time in the past. I think that's the wrong approach. It's a locally optimized approach that fails to ask the more interesting question. Is OpenStack better as a whole if this is a mandatory component of OpenStack? Better being defined as technically better (more features, less janky code work arounds, less unexpected behavior from the stack). Better from the sense of easier or harder to run an actual cloud by our Operators (taking into account what kinds of moving parts they are now expected to manage). Better from the sense of a better user experience in interacting with OpenStack as whole. Better from a sense that the OpenStack release will experience less bugs, less unexpected cross project interactions, an a greater overall feel of consistency so that the OpenStack API feels like one thing. https://dague.net/2014/08/26/openstack-as-layers/ One of the interesting qualities of Layers 1 2 is they all follow an AMQP + RDBMS pattern (excepting swift). You can have a very effective IaaS out of that stack. They are the things that you can provide pretty solid integration testing on (and if you look at where everything stood before the new TC mandates on testing / upgrade that was basically what was getting integration tested). (Also note, I'll accept Barbican is probably in the wrong layer, and should be a Layer 2 service.) While large shops can afford to have a dedicated team to figure out how to make mongo or redis HA, provide monitoring, have a DR plan for when a huricane requires them to flip datacenters, that basically means OpenStack heads further down the path of only for the big folks. I don't want OpenStack to be only for the big folks, I want OpenStack to be for all sized folks. I really do want to have all the local small colleges around
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
Tim Bell wrote: -Original Message- From: Thierry Carrez [mailto:thie...@openstack.org] Sent: 04 September 2014 16:59 To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting Sean Dague wrote: [...] So, honestly, I'll probably remain -1 on the final integration vote, not because Zaqar is bad, but because I'm feeling more firmly that for OpenStack to not leave the small deployers behind we need to redefine the tightly integrated piece of OpenStack to basically the Layer 1 2 parts of my diagram, and consider the rest of the layers exciting parts of our ecosystem that more advanced users may choose to deploy to meet their needs. Smaller tent, big ecosystem, easier on ramp. I realize that largely means Zaqar would be caught up in a definition discussion outside of it's control, and that's kind of unfortunate, as Flavio and team have been doing a bang up job of late. But we need to stop considering integration as the end game of all interesting software in the OpenStack ecosystem, and I think it's better to have that conversation sooner rather than later. I think it's pretty clear at this point that: (1) we need to have a discussion about layers (base nucleus, optional extra services at the very least) and the level of support we grant to each -- the current binary approach is not working very well (2) If we accept Zaqar next week, it's pretty clear it would not fall in the base nucleus layer but more in an optional extra services layer, together with at the very least Trove and Sahara There are two ways of doing this: follow Sean's approach and -1 integration (and have zaqar apply to that optional layer when we create it), or +1 integration now (and have zaqar follow whichever other integrated projects we place in that layer when we create it). I'm still hesitating on the best approach. I think they yield the same end result, but the -1 approach seems to be a bit more unfair, since it would be purely for reasons we don't (yet) apply to currently-integrated projects... The one concern I have with a small core is that there is not an easy way to assess the maturity of a project on stackforge. The stackforge projects may be missing packaging, Red Hat testing, puppet modules, install/admin documentation etc. Thus, I need to have some indication that a project is deployable before looking at it with my user community to see if it meets a need that is sustainable. Do you see the optional layer services being blessed / validated in some way and therefore being easy to identify ? Yes, I think whatever exact shape this takes, it should convey some assertion of stability to be able to distinguish itself from random projects. Some way of saying this is good and mature, even if it's not in the inner circle. Being in The integrated release has been seen as a sign of stability forever, while it was only ensuring integration with other projects and OpenStack processes. We are getting better at requiring maturity there, but if we set up layers, we'll have to get even better at that. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/04/2014 09:44 PM, Kurt Griffiths wrote: Thanks for your comments Gordon. I appreciate where you are coming from and I think we are actually in agreement on a lot of things. I just want to make it clear that from the very beginning of the project the team has tried to communicate (but perhaps could have done a better job at it) that we aren’t trying to displace other messaging systems that are clearly delivering a lot of value today. In fact, I personally have long been a proponent of using the best tool for the job. The Zaqar project was kicked off at an unconference session several summits ago because the community saw a need that was not covered by other messaging systems. Does that mean those other systems are “bad” or “wrong”? Of course not. It simply means that there are some cases where those other systems aren’t the best tool for the job, and another tool is needed (and vice versa). I think communicating that unmet need, those use-cases not best served by other systems, would help a lot in clarifying Zaqar's intended role. Does that other tool look *exactly* like Zaqar? Probably not. But a lot of people have told us Zaqar--in its current form--already delivers a lot of value that they can’t get from other messaging systems that are available to them. Zaqar, like any open source project, is a manifestation of lots of peoples' ideas, and will evolve over time to meet the needs of the community. Does a Qpid/Rabbit/Kafka provisioning service make sense? Probably. Would such a service totally overlap in terms of use-cases with Zaqar? Community feedback suggests otherwise. Will there be some other kind of thing that comes out of the woodwork? Possibly. (Heck, if something better comes along I for one have no qualms in shifting resources to the more elegant solution--again, use the best tool for the job.) This process happens all the time in the broader open-source world. But this process takes a healthy amount of time, plus broad exposure and usage, which is something that you simply don’t get as a non-integrated project in the OpenStack ecosystem. In any case, it’s pretty clear to me that Zaqar graduating should not be viewed as making it the officially blessed messaging service for the cloud” and nobody is allowed to have any other ideas, ever. Indeed, and to be clear, I wasn't really commenting on the graduation at all. I was really just responding to the statements on scope and differentiation; 'messaging service for the cloud' is a very broad problem space and as you rightly point out there may be different tools that best serve different parts of that problem space. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 5 September 2014 23:33, Sean Dague s...@dague.net wrote: I think realistically a self certification process that would have artifacts in a discoverable place. I was thinking something along the lines of a baseball card interface with a short description of the project, a list of the requirements to deploy (native and python), a link the the API docs, a link to current test coverage, as well as some statement on the frequency of testing, stats on open bugs and current trending and review backlog, current user testimonials. Basically the kind of first stage analysis that a deployer would do before pushing this out to their users. Add into that their deployment support - e.g. do they have TripleO support // Chef // Fuel // Puppet etc etc etc. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/04/2014 09:44 PM, Kurt Griffiths wrote: Does a Qpid/Rabbit/Kafka provisioning service make sense? Probably. I think something like that would be valuable, especially in conjunction with some application layer proxying and mapping between 'virtual' addresses/endpoints and specific queues/virtual hosts/brokers. That would allow people to use the brokers, protocols and semantics they are already familiar with, combine different systems even, and have self-provisioning and scalability on top of it, with varying degrees of isolation for tenants. Would such a service totally overlap in terms of use-cases with Zaqar? Community feedback suggests otherwise. I'd love to read the feedback if you have any links. While I agree overlap is rarely total, I suspect there will be quite a bit and that it will grow as each approach evolves. That emphatically does not mean I think Zaqar should in any way be held back though, just that I think the openstack community should anticipate and indeed encourage other approaches also. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/04/2014 01:14 PM, Sean Dague wrote: https://dague.net/2014/08/26/openstack-as-layers/ Just wanted to say that I found this article very useful indeed and agree with the points you make in it. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
2014-09-05 9:09 GMT-03:00 Sean Dague s...@dague.net: On 09/05/2014 07:39 AM, Robert Collins wrote: On 5 September 2014 23:33, Sean Dague s...@dague.net wrote: I think realistically a self certification process that would have artifacts in a discoverable place. I was thinking something along the lines of a baseball card interface with a short description of the project, a list of the requirements to deploy (native and python), a link the the API docs, a link to current test coverage, as well as some statement on the frequency of testing, stats on open bugs and current trending and review backlog, current user testimonials. Basically the kind of first stage analysis that a deployer would do before pushing this out to their users. Add into that their deployment support - e.g. do they have TripleO support // Chef // Fuel // Puppet etc etc etc. ACK, good points. I expect packaging might go on that list as well. I won't pretend I've got the whole baseball card self cert thing worked out, just trying to sketch an idea that might bear fruit. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Hi all, Thanks for bringing up this topic Flavio and everyone for the feedback! From my humble junior developer point of view I would like to share some comments about some of the concerns mentioned. - Added complexity on adding Zaqar to the ecosystem and the NoSQL concern I won't deny that adding Zaqar, if operators need it, will make OpenStack a little more harder to deploy. But this will happen with any extra project added. Complexity is part of OpenStack though. We, as OpenStack, are trying to provide a software solution for a really complex need. And that is what makes us great. IMO operators interested in using Zaqar will consider the pros and cons of adding it and will make their choice based on that. And it's not about the tools we use. NoSQL was chosen to make the job for Zaqar because it was proven to do a better job. And, in the whole family of NoSQL solutions, we choose the ones that were considered easier to deploy. It's a tecnology that has been in use for a long time now and it fits perfectly the requirements of Zaqar. In this regard, I think there is a long way to go and is something that Zaqar care about everyday. Zaqar will keep on researching for the best solutions for the users and working on adding support for them, as every other project does. - Reinventing or not reinventing a messaging system In the last couple of months I has been working on adding AMQP as a storage/transport backend for Zaqar. During that period I managed to learn a lot from other messaging systems, including the ones that has been discussed from now. With that basis I can say that Zaqar is covering other, different, uses cases that the mentioned technologies are not meant to cover. And the best of all, it's being specialized for the cloud. Most widely used cloud services already has they message and notification systems solution, and we should be up to the game with that. Right now it doesn't seems like that need is being filled. - Integration as a sign of stability Right now we are in a situation in which Zaqar is a mature project with many features to provide, but in order to keep on growing and getting better it needs to integrate with other projects and start showing what is capable of. Zaqar is robust, well documented and with a team willing to keep on enhancing it. It doesn't matter what places it takes on the stack, what is matter is that it's on the stack. Hope this all makes sense and please correct me if I'm wrong. I want the best for both OpenStack and Zaqar, as you do. All the best, Victoria ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On Fri, Sep 5, 2014 at 4:27 AM, Thierry Carrez thie...@openstack.org wrote: Tim Bell wrote: The one concern I have with a small core is that there is not an easy way to assess the maturity of a project on stackforge. The stackforge projects may be missing packaging, Red Hat testing, puppet modules, install/admin documentation etc. Thus, I need to have some indication that a project is deployable before looking at it with my user community to see if it meets a need that is sustainable. Do you see the optional layer services being blessed / validated in some way and therefore being easy to identify ? Yes, I think whatever exact shape this takes, it should convey some assertion of stability to be able to distinguish itself from random projects. Some way of saying this is good and mature, even if it's not in the inner circle. Being in The integrated release has been seen as a sign of stability forever, while it was only ensuring integration with other projects and OpenStack processes. We are getting better at requiring maturity there, but if we set up layers, we'll have to get even better at that. The layers are (or originally were) a purely technical organization, intentionally avoiding association with defcore and other groupings, and on reflection, maturity too. The problem that repeatedly bubbles up is that people (mostly outside the community) want a simple tag for maturity or blessedness and have been using the integrated/incubated status for that. Maturity has nothing to do with technical relationships between projects (required/optional layers). The good and mature blessing should be an independent attribute that is set on projects as a result of nomination by TC members or PTLs or existing core members or whatever trusted group we choose. I'd say for starters that anything from Stackforge that we use in integrated/incubated projects is on the short list for that status, as it already has that implicitly by our use. dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
If I take a recent case we had, we’re looking for ways to simplify backup for our users. They should not have to write crons to do snapshots but leave that to a service. Raksha seems promising… a wiki page on openstack.org, code in github etc. How can the average deployer know whether a stackforge is a. An early prototype which has completed (such as some of the early LBaaS packages) b. A project which has lost its initial steam and further investment is not foreseen c. A reliable project where there has not been a significant need to change recently but is a good long term bet The end result is that deployers hold back waiting for an indication that this is more than a prototype in whatever form that would take. This means a missed opportunity as if something is interesting to many deployers, that could create the community needed to keep a project going. The statement of “this is good and mature, even if it's not in the inner circle is exactly what is needed. When I offer something to my end-users, there is an implied promise of this (and a corresponding credibility issue if that is not met). Tim From: Dean Troyer [mailto:dtro...@gmail.com] Sent: 05 September 2014 19:11 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting On Fri, Sep 5, 2014 at 4:27 AM, Thierry Carrez thie...@openstack.orgmailto:thie...@openstack.org wrote: Tim Bell wrote: The one concern I have with a small core is that there is not an easy way to assess the maturity of a project on stackforge. The stackforge projects may be missing packaging, Red Hat testing, puppet modules, install/admin documentation etc. Thus, I need to have some indication that a project is deployable before looking at it with my user community to see if it meets a need that is sustainable. Do you see the optional layer services being blessed / validated in some way and therefore being easy to identify ? Yes, I think whatever exact shape this takes, it should convey some assertion of stability to be able to distinguish itself from random projects. Some way of saying this is good and mature, even if it's not in the inner circle. Being in The integrated release has been seen as a sign of stability forever, while it was only ensuring integration with other projects and OpenStack processes. We are getting better at requiring maturity there, but if we set up layers, we'll have to get even better at that. The layers are (or originally were) a purely technical organization, intentionally avoiding association with defcore and other groupings, and on reflection, maturity too. The problem that repeatedly bubbles up is that people (mostly outside the community) want a simple tag for maturity or blessedness and have been using the integrated/incubated status for that. Maturity has nothing to do with technical relationships between projects (required/optional layers). The good and mature blessing should be an independent attribute that is set on projects as a result of nomination by TC members or PTLs or existing core members or whatever trusted group we choose. I'd say for starters that anything from Stackforge that we use in integrated/incubated projects is on the short list for that status, as it already has that implicitly by our use. dt -- Dean Troyer dtro...@gmail.commailto:dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
Greetings, Last Tuesday the TC held the first graduation review for Zaqar. During the meeting some concerns arose. I've listed those concerns below with some comments hoping that it will help starting a discussion before the next meeting. In addition, I've added some comments about the project stability at the bottom and an etherpad link pointing to a list of use cases for Zaqar. # Concerns - Concern on operational burden of requiring NoSQL deploy expertise to the mix of openstack operational skills For those of you not familiar with Zaqar, it currently supports 2 nosql drivers - MongoDB and Redis - and those are the only 2 drivers it supports for now. This will require operators willing to use Zaqar to maintain a new (?) NoSQL technology in their system. Before expressing our thoughts on this matter, let me say that: 1. By removing the SQLAlchemy driver, we basically removed the chance for operators to use an already deployed OpenStack-technology 2. Zaqar won't be backed by any AMQP based messaging technology for now. Here's[0] a summary of the research the team (mostly done by Victoria) did during Juno 3. We (OpenStack) used to require Redis for the zmq matchmaker 4. We (OpenStack) also use memcached for caching and as the oslo caching lib becomes available - or a wrapper on top of dogpile.cache - Redis may be used in place of memcached in more and more deployments. 5. Ceilometer's recommended storage driver is still MongoDB, although Ceilometer has now support for sqlalchemy. (Please correct me if I'm wrong). That being said, it's obvious we already, to some extent, promote some NoSQL technologies. However, for the sake of the discussion, lets assume we don't. I truly believe, with my OpenStack (not Zaqar's) hat on, that we can't keep avoiding these technologies. NoSQL technologies have been around for years and we should be prepared - including OpenStack operators - to support these technologies. Not every tool is good for all tasks - one of the reasons we removed the sqlalchemy driver in the first place - therefore it's impossible to keep an homogeneous environment for all services. With this, I'm not suggesting to ignore the risks and the extra burden this adds but, instead of attempting to avoid it completely by not evolving the stack of services we provide, we should probably work on defining a reasonable subset of NoSQL services we are OK with supporting. This will help making the burden smaller and it'll give operators the option to choose. [0] http://blog.flaper87.com/post/marconi-amqp-see-you-later/ - Concern on should we really reinvent a queue system rather than piggyback on one As mentioned in the meeting on Tuesday, Zaqar is not reinventing message brokers. Zaqar provides a service akin to SQS from AWS with an OpenStack flavor on top. [0] Some things that differentiate Zaqar from SQS is it's capability for supporting different protocols without sacrificing multi-tenantcy and other intrinsic features it provides. Some protocols you may consider for Zaqar are: STOMP, MQTT. As far as the backend goes, Zaqar is not re-inventing it either. It sits on top of existing storage technologies that have proven to be fast and reliable for this task. The choice of using NoSQL technologies has a lot to do with this particular thing and the fact that Zaqar needs a storage capable of scaling, replicating and good support for failover. [0] https://wiki.openstack.org/wiki/Zaqar/Frequently_asked_questions#Is_Zaqar_a_provisioning_service_or_a_data_API.3F - concern on dataplane vs. controlplane, should we add more dataplane things in the integrated release I'm really not sure I understand the arguments against dataplane services. What concerns do people have about these services? As far as I can tell, we already have several services - some in the lower layers - that provide a data plane API. For example: * keystone (service catalogs and tokens) * glance (image management) * swift (object storage) * ceilometer (metrics) * heat (provisioning) * barbican (key management) Are the concerns specific to Zaqar's dataplane API? - concern on API v2 being already planned At the meeting, we discussed a bit about Zaqar's API and more importantly how stable it is. During that discussion I mentioned an hypothetical v2 of the API. I'd like to clarify that a v2 is not being planned for Kilo, what we would like to do is to gather feedback from the community and services consuming Zaqar about the existing API and use that feedback to design a new version of the API if necessary. All this has yet to be discussed but most importantly, we would first like to get more feedback from the community. We have already gotten some feedback, but it has been fairly limited because most people are waiting for us to graduate before kicking the tires. We do have some endpoints that will go away in the API v2 - getting messages by id,
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
Excerpts from Flavio Percoco's message of 2014-09-04 00:08:47 -0700: Greetings, Last Tuesday the TC held the first graduation review for Zaqar. During the meeting some concerns arose. I've listed those concerns below with some comments hoping that it will help starting a discussion before the next meeting. In addition, I've added some comments about the project stability at the bottom and an etherpad link pointing to a list of use cases for Zaqar. Hi Flavio. This was an interesting read. As somebody whose attention has recently been drawn to Zaqar, I am quite interested in seeing it graduate. # Concerns - Concern on operational burden of requiring NoSQL deploy expertise to the mix of openstack operational skills For those of you not familiar with Zaqar, it currently supports 2 nosql drivers - MongoDB and Redis - and those are the only 2 drivers it supports for now. This will require operators willing to use Zaqar to maintain a new (?) NoSQL technology in their system. Before expressing our thoughts on this matter, let me say that: 1. By removing the SQLAlchemy driver, we basically removed the chance for operators to use an already deployed OpenStack-technology 2. Zaqar won't be backed by any AMQP based messaging technology for now. Here's[0] a summary of the research the team (mostly done by Victoria) did during Juno 3. We (OpenStack) used to require Redis for the zmq matchmaker 4. We (OpenStack) also use memcached for caching and as the oslo caching lib becomes available - or a wrapper on top of dogpile.cache - Redis may be used in place of memcached in more and more deployments. 5. Ceilometer's recommended storage driver is still MongoDB, although Ceilometer has now support for sqlalchemy. (Please correct me if I'm wrong). That being said, it's obvious we already, to some extent, promote some NoSQL technologies. However, for the sake of the discussion, lets assume we don't. I truly believe, with my OpenStack (not Zaqar's) hat on, that we can't keep avoiding these technologies. NoSQL technologies have been around for years and we should be prepared - including OpenStack operators - to support these technologies. Not every tool is good for all tasks - one of the reasons we removed the sqlalchemy driver in the first place - therefore it's impossible to keep an homogeneous environment for all services. I whole heartedly agree that non traditional storage technologies that are becoming mainstream are good candidates for use cases where SQL based storage gets in the way. I wish there wasn't so much FUD (warranted or not) about MongoDB, but that is the reality we live in. With this, I'm not suggesting to ignore the risks and the extra burden this adds but, instead of attempting to avoid it completely by not evolving the stack of services we provide, we should probably work on defining a reasonable subset of NoSQL services we are OK with supporting. This will help making the burden smaller and it'll give operators the option to choose. [0] http://blog.flaper87.com/post/marconi-amqp-see-you-later/ - Concern on should we really reinvent a queue system rather than piggyback on one As mentioned in the meeting on Tuesday, Zaqar is not reinventing message brokers. Zaqar provides a service akin to SQS from AWS with an OpenStack flavor on top. [0] I think Zaqar is more like SMTP and IMAP than AMQP. You're not really trying to connect two processes in real time. You're trying to do fully asynchronous messaging with fully randomized access to any message. Perhaps somebody should explore whether the approaches taken by large scale IMAP providers could be applied to Zaqar. Anyway, I can't imagine writing a system to intentionally use the semantics of IMAP and SMTP. I'd be very interested in seeing actual use cases for it, apologies if those have been posted before. Some things that differentiate Zaqar from SQS is it's capability for supporting different protocols without sacrificing multi-tenantcy and other intrinsic features it provides. Some protocols you may consider for Zaqar are: STOMP, MQTT. As far as the backend goes, Zaqar is not re-inventing it either. It sits on top of existing storage technologies that have proven to be fast and reliable for this task. The choice of using NoSQL technologies has a lot to do with this particular thing and the fact that Zaqar needs a storage capable of scaling, replicating and good support for failover. What's odd to me is that other systems like Cassandra and Riak are not being discussed. There are well documented large scale message storage systems on both, and neither is encumbered by the same licensing FUD as MongoDB. Anyway, again if we look at this as a place to storage and retrieve messages, and not as a queue, then talking about databases, instead of message brokers, makes a lot more sense. - concern on the maturity of the NoQSL not AGPL backend (Redis)
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
Hey Clint, Thanks for reading, some comments in-line: On 09/04/2014 10:30 AM, Clint Byrum wrote: Excerpts from Flavio Percoco's message of 2014-09-04 00:08:47 -0700: [snip] - Concern on should we really reinvent a queue system rather than piggyback on one As mentioned in the meeting on Tuesday, Zaqar is not reinventing message brokers. Zaqar provides a service akin to SQS from AWS with an OpenStack flavor on top. [0] I think Zaqar is more like SMTP and IMAP than AMQP. You're not really trying to connect two processes in real time. You're trying to do fully asynchronous messaging with fully randomized access to any message. Perhaps somebody should explore whether the approaches taken by large scale IMAP providers could be applied to Zaqar. Anyway, I can't imagine writing a system to intentionally use the semantics of IMAP and SMTP. I'd be very interested in seeing actual use cases for it, apologies if those have been posted before. Some things that differentiate Zaqar from SQS is it's capability for supporting different protocols without sacrificing multi-tenantcy and other intrinsic features it provides. Some protocols you may consider for Zaqar are: STOMP, MQTT. As far as the backend goes, Zaqar is not re-inventing it either. It sits on top of existing storage technologies that have proven to be fast and reliable for this task. The choice of using NoSQL technologies has a lot to do with this particular thing and the fact that Zaqar needs a storage capable of scaling, replicating and good support for failover. What's odd to me is that other systems like Cassandra and Riak are not being discussed. There are well documented large scale message storage systems on both, and neither is encumbered by the same licensing FUD as MongoDB. FWIW, they both have been discussed. As far as Cassandra goes, we raised the red flag after reading reading this post[0]. The post itself may be obsolete already but I don't think I have enough knowledge about Cassandra to actually figure this out. Some folks have come to us asking for a Cassandra driver and they were interested in contributing/working on one. I really hope that will happen someday, although it'll certainly happen as an external driver. Riak, on the other hand, was certainly a good candidate. What made us go with MongoDB and Redis is they're both good for the job, they are both likely already deployed in OpenStack clouds and we have enough knowledge to provide support and maintenance for both drivers. As a curious note, ElasticSearch and Swift have also been brought up several times as valid stores for Zaqar. I haven't thought about this throughly but I think they'd do a good job. [0] http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets Anyway, again if we look at this as a place to storage and retrieve messages, and not as a queue, then talking about databases, instead of message brokers, makes a lot more sense. - concern on the maturity of the NoQSL not AGPL backend (Redis) Redis backend just landed and I've been working on a gate job for it today. Although it hasn't been tested in production, if Zaqar graduates, it still has a full development cycle to be tested and improved before the first integrated release happens. I'd be quite interested to see how it is expected to scale. From my very quick reading of the driver, it only supports a single redis server. No consistent hash ring or anything like that. Indeed, support for redis-cluster is in our roadmap[0]. As of now, it can be scaled by using Zaqar pools. You can create several pools of redis nodes, that you can balance between queues. The next series of benchmarks will be done on this new Redis driver. I hope those will be ready soon. [0] https://blueprints.launchpad.net/zaqar/+spec/redis-pool # Use Cases In addition to the aforementioned concerns and comments, I also would like to share an etherpad that contains some use cases that other integrated projects have for Zaqar[0]. The list is not exhaustive and it'll contain more information before the next meeting. [0] https://etherpad.openstack.org/p/zaqar-integrated-projects-use-cases Just taking a look, there are two basic applications needed: 1) An inbox. Horizon wants to know when snapshots are done. Heat wants to know what happened during a stack action. Etc. 2) A user-focused message queue. Heat wants to push data to agents. Swift wants to synchronize processes when things happen. To me, #1 is Zaqar as it is today. #2 is the one that I worry may not be served best by bending #1 onto it. Push semantics are being developed. We've had enough discussions that have helped preparing the ground for it. However, I believe both use cases could be covered by Zaqar as-is. Could you elaborate a bit more on #2? Especially on why you think Zaqar as is can't serve this specific case? Also, feel free to add use-cases to that etherpad if
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On Thu, 4 Sep 2014, Flavio Percoco wrote: Thanks for writing this up, interesting read. 5. Ceilometer's recommended storage driver is still MongoDB, although Ceilometer has now support for sqlalchemy. (Please correct me if I'm wrong). For sake of reference: Yes, MongoDB is currently the recommended store and yes, sqlalchemy support is present. Until recently only sqlalchemy support was tested in the gate. Two big changes being developed in Juno related to storage: * Improved read and write performance in the sqlalchemy setup. * time series storage and Gnocchi: https://julien.danjou.info/blog/2014/openstack-ceilometer-the-gnocchi-experiment I truly believe, with my OpenStack (not Zaqar's) hat on, that we can't keep avoiding these technologies. NoSQL technologies have been around for years and we should be prepared - including OpenStack operators - to support these technologies. Not every tool is good for all tasks - one of the reasons we removed the sqlalchemy driver in the first place - therefore it's impossible to keep an homogeneous environment for all services. +1. Ain't that the truth. As mentioned in the meeting on Tuesday, Zaqar is not reinventing message brokers. Zaqar provides a service akin to SQS from AWS with an OpenStack flavor on top. [0] In my efforts to track this stuff I remain confused on the points in these two questions: https://wiki.openstack.org/wiki/Zaqar/Frequently_asked_questions#How_does_Zaqar_compare_to_oslo.messaging.3F https://wiki.openstack.org/wiki/Zaqar/Frequently_asked_questions#Is_Zaqar_an_under-cloud_or_an_over-cloud_service.3F What or where is the boundary between Zaqar and existing messaging infrastructure? Not just in terms of technology but also use cases? The answers above suggest its not super solid on the use case side, notably: In addition, several projects have expressed interest in integrating with Zaqar in order to surface events... Instead of Zaqar doing what it does and instead of oslo.messaging abstracting RPC, why isn't the end goal a multi-tenant, multi-protocol event pool? Wouldn't that have the most flexibility in terms of ecosystem and scalability? In addition to the aforementioned concerns and comments, I also would like to share an etherpad that contains some use cases that other integrated projects have for Zaqar[0]. The list is not exhaustive and it'll contain more information before the next meeting. [0] https://etherpad.openstack.org/p/zaqar-integrated-projects-use-cases For these, what is Zaqar providing that oslo.messaging (and its still extant antecedents) does not? I'm not asking to naysay Zaqar, but to understand more clearly what's going on. My interest here comes from a general interest in now events and notifications are handled throughout OpenStack. Thanks. -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/04/2014 03:08 AM, Flavio Percoco wrote: Greetings, Last Tuesday the TC held the first graduation review for Zaqar. During the meeting some concerns arose. I've listed those concerns below with some comments hoping that it will help starting a discussion before the next meeting. In addition, I've added some comments about the project stability at the bottom and an etherpad link pointing to a list of use cases for Zaqar. # Concerns - Concern on operational burden of requiring NoSQL deploy expertise to the mix of openstack operational skills For those of you not familiar with Zaqar, it currently supports 2 nosql drivers - MongoDB and Redis - and those are the only 2 drivers it supports for now. This will require operators willing to use Zaqar to maintain a new (?) NoSQL technology in their system. Before expressing our thoughts on this matter, let me say that: 1. By removing the SQLAlchemy driver, we basically removed the chance for operators to use an already deployed OpenStack-technology 2. Zaqar won't be backed by any AMQP based messaging technology for now. Here's[0] a summary of the research the team (mostly done by Victoria) did during Juno 3. We (OpenStack) used to require Redis for the zmq matchmaker 4. We (OpenStack) also use memcached for caching and as the oslo caching lib becomes available - or a wrapper on top of dogpile.cache - Redis may be used in place of memcached in more and more deployments. 5. Ceilometer's recommended storage driver is still MongoDB, although Ceilometer has now support for sqlalchemy. (Please correct me if I'm wrong). That being said, it's obvious we already, to some extent, promote some NoSQL technologies. However, for the sake of the discussion, lets assume we don't. I truly believe, with my OpenStack (not Zaqar's) hat on, that we can't keep avoiding these technologies. NoSQL technologies have been around for years and we should be prepared - including OpenStack operators - to support these technologies. Not every tool is good for all tasks - one of the reasons we removed the sqlalchemy driver in the first place - therefore it's impossible to keep an homogeneous environment for all services. With this, I'm not suggesting to ignore the risks and the extra burden this adds but, instead of attempting to avoid it completely by not evolving the stack of services we provide, we should probably work on defining a reasonable subset of NoSQL services we are OK with supporting. This will help making the burden smaller and it'll give operators the option to choose. [0] http://blog.flaper87.com/post/marconi-amqp-see-you-later/ I've been one of the consistent voices concerned about a hard requirement on adding NoSQL into the mix. So I'll explain that thinking a bit more. I feel like when the TC makes an integration decision previously this has been about evaluating the project applying for integration, and if they met some specific criteria they were told about some time in the past. I think that's the wrong approach. It's a locally optimized approach that fails to ask the more interesting question. Is OpenStack better as a whole if this is a mandatory component of OpenStack? Better being defined as technically better (more features, less janky code work arounds, less unexpected behavior from the stack). Better from the sense of easier or harder to run an actual cloud by our Operators (taking into account what kinds of moving parts they are now expected to manage). Better from the sense of a better user experience in interacting with OpenStack as whole. Better from a sense that the OpenStack release will experience less bugs, less unexpected cross project interactions, an a greater overall feel of consistency so that the OpenStack API feels like one thing. https://dague.net/2014/08/26/openstack-as-layers/ One of the interesting qualities of Layers 1 2 is they all follow an AMQP + RDBMS pattern (excepting swift). You can have a very effective IaaS out of that stack. They are the things that you can provide pretty solid integration testing on (and if you look at where everything stood before the new TC mandates on testing / upgrade that was basically what was getting integration tested). (Also note, I'll accept Barbican is probably in the wrong layer, and should be a Layer 2 service.) While large shops can afford to have a dedicated team to figure out how to make mongo or redis HA, provide monitoring, have a DR plan for when a huricane requires them to flip datacenters, that basically means OpenStack heads further down the path of only for the big folks. I don't want OpenStack to be only for the big folks, I want OpenStack to be for all sized folks. I really do want to have all the local small colleges around here have OpenStack clouds, because it's something that people believe they can do and manage. I know the people that work in this places, they all come out to the LUG I run.
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/04/2014 01:15 PM, Chris Dent wrote: On Thu, 4 Sep 2014, Flavio Percoco wrote: Thanks for writing this up, interesting read. Thank you for your feedback :) Some comments in-line. 5. Ceilometer's recommended storage driver is still MongoDB, although Ceilometer has now support for sqlalchemy. (Please correct me if I'm wrong). For sake of reference: Yes, MongoDB is currently the recommended store and yes, sqlalchemy support is present. Until recently only sqlalchemy support was tested in the gate. Two big changes being developed in Juno related to storage: * Improved read and write performance in the sqlalchemy setup. * time series storage and Gnocchi: https://julien.danjou.info/blog/2014/openstack-ceilometer-the-gnocchi-experiment Awesome, thanks for clarifying this. [snip] As mentioned in the meeting on Tuesday, Zaqar is not reinventing message brokers. Zaqar provides a service akin to SQS from AWS with an OpenStack flavor on top. [0] In my efforts to track this stuff I remain confused on the points in these two questions: https://wiki.openstack.org/wiki/Zaqar/Frequently_asked_questions#How_does_Zaqar_compare_to_oslo.messaging.3F https://wiki.openstack.org/wiki/Zaqar/Frequently_asked_questions#Is_Zaqar_an_under-cloud_or_an_over-cloud_service.3F What or where is the boundary between Zaqar and existing messaging infrastructure? Not just in terms of technology but also use cases? The answers above suggest its not super solid on the use case side, notably: In addition, several projects have expressed interest in integrating with Zaqar in order to surface events... Instead of Zaqar doing what it does and instead of oslo.messaging abstracting RPC, why isn't the end goal a multi-tenant, multi-protocol event pool? Wouldn't that have the most flexibility in terms of ecosystem and scalability? If we put both features, multi-tenancy and multi-protocol, aside for a bit, we can simplify Zaqars goal down to a messaging service for the cloud. I believe this is exactly where the line between Zaqar and other *queuing* technologies should be drawn. Zaqar is, at the very end, a messaging service thought for the cloud whereas existing queuing technologies were not designed for it. By cloud I don't mean performance, scalability nor anything like that. I'm talking about providing a service that end-users of the cloud can consume. The fact that Zaqar is also ideal for the under-cloud is a plus. The service has been designed to suffice a set of messaging features that serve perfectly use-cases in both the under-cloud and over-cloud. If we add to that a multi-protocol transport layer with support for multi-tenancy, you'll get a queuing service that fits the need of cloud providers and covers a broader set of use cases like, say, IoT. I forgot to add this link[0] to my previous email. Does the overview of the service, the key features and scope help clearing things out a bit? Please, let me know if they don't. I'm happy to provide more info if needed. [0] https://wiki.openstack.org/wiki/Zaqar#Overview In addition to the aforementioned concerns and comments, I also would like to share an etherpad that contains some use cases that other integrated projects have for Zaqar[0]. The list is not exhaustive and it'll contain more information before the next meeting. [0] https://etherpad.openstack.org/p/zaqar-integrated-projects-use-cases For these, what is Zaqar providing that oslo.messaging (and its still extant antecedents) does not? I'm not asking to naysay Zaqar, but to understand more clearly what's going on. My interest here comes from a general interest in now events and notifications are handled throughout OpenStack. One of the reasons you would want to use Zaqar instead of oslo.messaging for, say, guest-agents is that you don't want guest-agents to talk to your main messaging layer. Zaqar will help guest-agents to communicate with the main service in a more secure, authenticated and isolated way. If you were going to do that with oslo.messaging, you'd need to have separate virtual_hosts, exchanges and probably even users. This things cannot be easily configured without manual intervention. With Zaqar you can easily rely on your deployed cloud services - keystone, Barbican and Zaqar, for example - to achieve such isolation and security. There are also other aspects that are worrisome of relying on the main messaging infrastructure for the use cases mentioned in that etherpad. For example, using OpenStack's main rabbitmq instance to communicate with guest-agents would increase the workload on the infrastructure, which would require a better scaling strategy for it. I hope the above clears your doubts. Thanks a lot for your feedback, it's useful to keep the discussion going and helps everyone to keep re-evaluating the goals and scopes of the project. I hope other folks from the team will also chime in and share their thoughts. Cheers, Flavio --
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/04/2014 02:14 PM, Sean Dague wrote: On 09/04/2014 03:08 AM, Flavio Percoco wrote: Greetings, Last Tuesday the TC held the first graduation review for Zaqar. During the meeting some concerns arose. I've listed those concerns below with some comments hoping that it will help starting a discussion before the next meeting. In addition, I've added some comments about the project stability at the bottom and an etherpad link pointing to a list of use cases for Zaqar. # Concerns - Concern on operational burden of requiring NoSQL deploy expertise to the mix of openstack operational skills For those of you not familiar with Zaqar, it currently supports 2 nosql drivers - MongoDB and Redis - and those are the only 2 drivers it supports for now. This will require operators willing to use Zaqar to maintain a new (?) NoSQL technology in their system. Before expressing our thoughts on this matter, let me say that: 1. By removing the SQLAlchemy driver, we basically removed the chance for operators to use an already deployed OpenStack-technology 2. Zaqar won't be backed by any AMQP based messaging technology for now. Here's[0] a summary of the research the team (mostly done by Victoria) did during Juno 3. We (OpenStack) used to require Redis for the zmq matchmaker 4. We (OpenStack) also use memcached for caching and as the oslo caching lib becomes available - or a wrapper on top of dogpile.cache - Redis may be used in place of memcached in more and more deployments. 5. Ceilometer's recommended storage driver is still MongoDB, although Ceilometer has now support for sqlalchemy. (Please correct me if I'm wrong). That being said, it's obvious we already, to some extent, promote some NoSQL technologies. However, for the sake of the discussion, lets assume we don't. I truly believe, with my OpenStack (not Zaqar's) hat on, that we can't keep avoiding these technologies. NoSQL technologies have been around for years and we should be prepared - including OpenStack operators - to support these technologies. Not every tool is good for all tasks - one of the reasons we removed the sqlalchemy driver in the first place - therefore it's impossible to keep an homogeneous environment for all services. With this, I'm not suggesting to ignore the risks and the extra burden this adds but, instead of attempting to avoid it completely by not evolving the stack of services we provide, we should probably work on defining a reasonable subset of NoSQL services we are OK with supporting. This will help making the burden smaller and it'll give operators the option to choose. [0] http://blog.flaper87.com/post/marconi-amqp-see-you-later/ I've been one of the consistent voices concerned about a hard requirement on adding NoSQL into the mix. So I'll explain that thinking a bit more. I feel like when the TC makes an integration decision previously this has been about evaluating the project applying for integration, and if they met some specific criteria they were told about some time in the past. I think that's the wrong approach. It's a locally optimized approach that fails to ask the more interesting question. Is OpenStack better as a whole if this is a mandatory component of OpenStack? Better being defined as technically better (more features, less janky code work arounds, less unexpected behavior from the stack). Better from the sense of easier or harder to run an actual cloud by our Operators (taking into account what kinds of moving parts they are now expected to manage). Better from the sense of a better user experience in interacting with OpenStack as whole. Better from a sense that the OpenStack release will experience less bugs, less unexpected cross project interactions, an a greater overall feel of consistency so that the OpenStack API feels like one thing. https://dague.net/2014/08/26/openstack-as-layers/ One of the interesting qualities of Layers 1 2 is they all follow an AMQP + RDBMS pattern (excepting swift). You can have a very effective IaaS out of that stack. They are the things that you can provide pretty solid integration testing on (and if you look at where everything stood before the new TC mandates on testing / upgrade that was basically what was getting integration tested). (Also note, I'll accept Barbican is probably in the wrong layer, and should be a Layer 2 service.) While large shops can afford to have a dedicated team to figure out how to make mongo or redis HA, provide monitoring, have a DR plan for when a huricane requires them to flip datacenters, that basically means OpenStack heads further down the path of only for the big folks. I don't want OpenStack to be only for the big folks, I want OpenStack to be for all sized folks. I really do want to have all the local small colleges around here have OpenStack clouds, because it's something that people believe they can do and manage. I know the
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
Sean Dague wrote: [...] So, honestly, I'll probably remain -1 on the final integration vote, not because Zaqar is bad, but because I'm feeling more firmly that for OpenStack to not leave the small deployers behind we need to redefine the tightly integrated piece of OpenStack to basically the Layer 1 2 parts of my diagram, and consider the rest of the layers exciting parts of our ecosystem that more advanced users may choose to deploy to meet their needs. Smaller tent, big ecosystem, easier on ramp. I realize that largely means Zaqar would be caught up in a definition discussion outside of it's control, and that's kind of unfortunate, as Flavio and team have been doing a bang up job of late. But we need to stop considering integration as the end game of all interesting software in the OpenStack ecosystem, and I think it's better to have that conversation sooner rather than later. I think it's pretty clear at this point that: (1) we need to have a discussion about layers (base nucleus, optional extra services at the very least) and the level of support we grant to each -- the current binary approach is not working very well (2) If we accept Zaqar next week, it's pretty clear it would not fall in the base nucleus layer but more in an optional extra services layer, together with at the very least Trove and Sahara There are two ways of doing this: follow Sean's approach and -1 integration (and have zaqar apply to that optional layer when we create it), or +1 integration now (and have zaqar follow whichever other integrated projects we place in that layer when we create it). I'm still hesitating on the best approach. I think they yield the same end result, but the -1 approach seems to be a bit more unfair, since it would be purely for reasons we don't (yet) apply to currently-integrated projects... -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On 09/04/2014 04:59 PM, Thierry Carrez wrote: Sean Dague wrote: [...] So, honestly, I'll probably remain -1 on the final integration vote, not because Zaqar is bad, but because I'm feeling more firmly that for OpenStack to not leave the small deployers behind we need to redefine the tightly integrated piece of OpenStack to basically the Layer 1 2 parts of my diagram, and consider the rest of the layers exciting parts of our ecosystem that more advanced users may choose to deploy to meet their needs. Smaller tent, big ecosystem, easier on ramp. I realize that largely means Zaqar would be caught up in a definition discussion outside of it's control, and that's kind of unfortunate, as Flavio and team have been doing a bang up job of late. But we need to stop considering integration as the end game of all interesting software in the OpenStack ecosystem, and I think it's better to have that conversation sooner rather than later. I think it's pretty clear at this point that: (1) we need to have a discussion about layers (base nucleus, optional extra services at the very least) and the level of support we grant to each -- the current binary approach is not working very well (2) If we accept Zaqar next week, it's pretty clear it would not fall in the base nucleus layer but more in an optional extra services layer, together with at the very least Trove and Sahara There are two ways of doing this: follow Sean's approach and -1 integration (and have zaqar apply to that optional layer when we create it), or +1 integration now (and have zaqar follow whichever other integrated projects we place in that layer when we create it). As I mentioned in my reply to Sean's email, I believe +1 integration is the correct thing to do. I know it's hard to believe that I'm saying this with my OpenStack hat on and not Zaqar's but that's the truth. I truly believe we can't stop OpenStack's growth on this. We'll manage these growth details later on as we've done so far. Growing is as important as managing the growth. Though, in this case we're not growing without any clue of what will happen. We've a well known path that all integrated projects have followed and, in this specific case, Zaqar is following. Re-evaluating projects is something that has happened - and should happen - every once in a while. Once we have a place for this optional services, we will have to re-evaluate all the integrated projects and move those that fit into that category. I'm still hesitating on the best approach. I think they yield the same end result, but the -1 approach seems to be a bit more unfair, since it would be purely for reasons we don't (yet) apply to currently-integrated projects... +1 Cheers, Flavio -- @flaper87 Flavio Percoco ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
Excerpts from Flavio Percoco's message of 2014-09-04 06:01:45 -0700: On 09/04/2014 02:14 PM, Sean Dague wrote: On 09/04/2014 03:08 AM, Flavio Percoco wrote: Greetings, Last Tuesday the TC held the first graduation review for Zaqar. During the meeting some concerns arose. I've listed those concerns below with some comments hoping that it will help starting a discussion before the next meeting. In addition, I've added some comments about the project stability at the bottom and an etherpad link pointing to a list of use cases for Zaqar. # Concerns - Concern on operational burden of requiring NoSQL deploy expertise to the mix of openstack operational skills For those of you not familiar with Zaqar, it currently supports 2 nosql drivers - MongoDB and Redis - and those are the only 2 drivers it supports for now. This will require operators willing to use Zaqar to maintain a new (?) NoSQL technology in their system. Before expressing our thoughts on this matter, let me say that: 1. By removing the SQLAlchemy driver, we basically removed the chance for operators to use an already deployed OpenStack-technology 2. Zaqar won't be backed by any AMQP based messaging technology for now. Here's[0] a summary of the research the team (mostly done by Victoria) did during Juno 3. We (OpenStack) used to require Redis for the zmq matchmaker 4. We (OpenStack) also use memcached for caching and as the oslo caching lib becomes available - or a wrapper on top of dogpile.cache - Redis may be used in place of memcached in more and more deployments. 5. Ceilometer's recommended storage driver is still MongoDB, although Ceilometer has now support for sqlalchemy. (Please correct me if I'm wrong). That being said, it's obvious we already, to some extent, promote some NoSQL technologies. However, for the sake of the discussion, lets assume we don't. I truly believe, with my OpenStack (not Zaqar's) hat on, that we can't keep avoiding these technologies. NoSQL technologies have been around for years and we should be prepared - including OpenStack operators - to support these technologies. Not every tool is good for all tasks - one of the reasons we removed the sqlalchemy driver in the first place - therefore it's impossible to keep an homogeneous environment for all services. With this, I'm not suggesting to ignore the risks and the extra burden this adds but, instead of attempting to avoid it completely by not evolving the stack of services we provide, we should probably work on defining a reasonable subset of NoSQL services we are OK with supporting. This will help making the burden smaller and it'll give operators the option to choose. [0] http://blog.flaper87.com/post/marconi-amqp-see-you-later/ I've been one of the consistent voices concerned about a hard requirement on adding NoSQL into the mix. So I'll explain that thinking a bit more. I feel like when the TC makes an integration decision previously this has been about evaluating the project applying for integration, and if they met some specific criteria they were told about some time in the past. I think that's the wrong approach. It's a locally optimized approach that fails to ask the more interesting question. Is OpenStack better as a whole if this is a mandatory component of OpenStack? Better being defined as technically better (more features, less janky code work arounds, less unexpected behavior from the stack). Better from the sense of easier or harder to run an actual cloud by our Operators (taking into account what kinds of moving parts they are now expected to manage). Better from the sense of a better user experience in interacting with OpenStack as whole. Better from a sense that the OpenStack release will experience less bugs, less unexpected cross project interactions, an a greater overall feel of consistency so that the OpenStack API feels like one thing. https://dague.net/2014/08/26/openstack-as-layers/ One of the interesting qualities of Layers 1 2 is they all follow an AMQP + RDBMS pattern (excepting swift). You can have a very effective IaaS out of that stack. They are the things that you can provide pretty solid integration testing on (and if you look at where everything stood before the new TC mandates on testing / upgrade that was basically what was getting integration tested). (Also note, I'll accept Barbican is probably in the wrong layer, and should be a Layer 2 service.) While large shops can afford to have a dedicated team to figure out how to make mongo or redis HA, provide monitoring, have a DR plan for when a huricane requires them to flip datacenters, that basically means OpenStack heads further down the path of only for the big folks. I don't want OpenStack to be only for the big folks, I want OpenStack to be for all sized folks. I really do
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
Excerpts from Flavio Percoco's message of 2014-09-04 02:11:15 -0700: Hey Clint, Thanks for reading, some comments in-line: On 09/04/2014 10:30 AM, Clint Byrum wrote: Excerpts from Flavio Percoco's message of 2014-09-04 00:08:47 -0700: [snip] - Concern on should we really reinvent a queue system rather than piggyback on one As mentioned in the meeting on Tuesday, Zaqar is not reinventing message brokers. Zaqar provides a service akin to SQS from AWS with an OpenStack flavor on top. [0] I think Zaqar is more like SMTP and IMAP than AMQP. You're not really trying to connect two processes in real time. You're trying to do fully asynchronous messaging with fully randomized access to any message. Perhaps somebody should explore whether the approaches taken by large scale IMAP providers could be applied to Zaqar. Anyway, I can't imagine writing a system to intentionally use the semantics of IMAP and SMTP. I'd be very interested in seeing actual use cases for it, apologies if those have been posted before. Some things that differentiate Zaqar from SQS is it's capability for supporting different protocols without sacrificing multi-tenantcy and other intrinsic features it provides. Some protocols you may consider for Zaqar are: STOMP, MQTT. As far as the backend goes, Zaqar is not re-inventing it either. It sits on top of existing storage technologies that have proven to be fast and reliable for this task. The choice of using NoSQL technologies has a lot to do with this particular thing and the fact that Zaqar needs a storage capable of scaling, replicating and good support for failover. What's odd to me is that other systems like Cassandra and Riak are not being discussed. There are well documented large scale message storage systems on both, and neither is encumbered by the same licensing FUD as MongoDB. FWIW, they both have been discussed. As far as Cassandra goes, we raised the red flag after reading reading this post[0]. The post itself may be obsolete already but I don't think I have enough knowledge about Cassandra to actually figure this out. Some folks have come to us asking for a Cassandra driver and they were interested in contributing/working on one. I really hope that will happen someday, although it'll certainly happen as an external driver. Riak, on the other hand, was certainly a good candidate. What made us go with MongoDB and Redis is they're both good for the job, they are both likely already deployed in OpenStack clouds and we have enough knowledge to provide support and maintenance for both drivers. It seems like Cassandra is good for when you're going to be writing all the time but only reading once. I would agree that this makes it less attractive for a generalized messaging platform, since you won't know how users will consume the messages, and if they are constantly reading then you'll have terrible performance. # Use Cases In addition to the aforementioned concerns and comments, I also would like to share an etherpad that contains some use cases that other integrated projects have for Zaqar[0]. The list is not exhaustive and it'll contain more information before the next meeting. [0] https://etherpad.openstack.org/p/zaqar-integrated-projects-use-cases Just taking a look, there are two basic applications needed: 1) An inbox. Horizon wants to know when snapshots are done. Heat wants to know what happened during a stack action. Etc. 2) A user-focused message queue. Heat wants to push data to agents. Swift wants to synchronize processes when things happen. To me, #1 is Zaqar as it is today. #2 is the one that I worry may not be served best by bending #1 onto it. Push semantics are being developed. We've had enough discussions that have helped preparing the ground for it. However, I believe both use cases could be covered by Zaqar as-is. Could you elaborate a bit more on #2? Especially on why you think Zaqar as is can't serve this specific case? The difference between 1 and 2 is that 2 is a true queueing problem. The message should go away when it has been consumed, and the volume may be rather high. With 1, you have a storage problem, and a database makes a lot more sense. If users can stick to type 2 problems, they'll be able to stay much more lightweight because they won't need a large data store that supports random access. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
Hi Flavio, On 09/04/2014 08:08 AM, Flavio Percoco wrote: - Concern on should we really reinvent a queue system rather than piggyback on one As mentioned in the meeting on Tuesday, Zaqar is not reinventing message brokers. Zaqar provides a service akin to SQS from AWS with an OpenStack flavor on top. [0] On 09/04/2014 01:13 PM, Flavio Percoco wrote: If we put both features, multi-tenancy and multi-protocol, aside for a bit, we can simplify Zaqars goal down to a messaging service for the cloud. I believe this is exactly where the line between Zaqar and other *queuing* technologies should be drawn. Zaqar is, at the very end, a messaging service thought for the cloud whereas existing queuing technologies were not designed for it. Isn't this the real answer to the concern above? Zaqar *is* a reimplementation of a messaging service, but you believe it is *necessary* to do this because existing implementations would not lend themselves to being as well integrated into the 'cloud', or do not provide sufficient support for multi-tenancy. I suspect if you asked, you would find that many existing projects implementing a messaging service would see their solution as very much suitable for the cloud. You disagree and probably with good reason. I'm just pointing out the dividing line is I think not as clear to everyone as it may be to you. If you add support for STOMP, MQTT, AMQP or any other protocol the overlap with other messaging services will be ever greater (and will bring in new requirements and new expectations around performance). This is not intended as criticism of Zaqar in anyway. In my opinion, 'reinvention' is not necessarily a bad thing. It's just another way of saying innovative approach and/or original thinking, both of which are good and both of which I think are important in the context of communicating in the cloud. My concern would be more that Zaqar's privileged status as the officially 'blessed' messaging service for the cloud - which is a broad space, much broader than just an SQS equivalent - would make it harder for other approaches to gain traction. --Gordon. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
-Original Message- From: Thierry Carrez [mailto:thie...@openstack.org] Sent: 04 September 2014 16:59 To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting Sean Dague wrote: [...] So, honestly, I'll probably remain -1 on the final integration vote, not because Zaqar is bad, but because I'm feeling more firmly that for OpenStack to not leave the small deployers behind we need to redefine the tightly integrated piece of OpenStack to basically the Layer 1 2 parts of my diagram, and consider the rest of the layers exciting parts of our ecosystem that more advanced users may choose to deploy to meet their needs. Smaller tent, big ecosystem, easier on ramp. I realize that largely means Zaqar would be caught up in a definition discussion outside of it's control, and that's kind of unfortunate, as Flavio and team have been doing a bang up job of late. But we need to stop considering integration as the end game of all interesting software in the OpenStack ecosystem, and I think it's better to have that conversation sooner rather than later. I think it's pretty clear at this point that: (1) we need to have a discussion about layers (base nucleus, optional extra services at the very least) and the level of support we grant to each -- the current binary approach is not working very well (2) If we accept Zaqar next week, it's pretty clear it would not fall in the base nucleus layer but more in an optional extra services layer, together with at the very least Trove and Sahara There are two ways of doing this: follow Sean's approach and -1 integration (and have zaqar apply to that optional layer when we create it), or +1 integration now (and have zaqar follow whichever other integrated projects we place in that layer when we create it). I'm still hesitating on the best approach. I think they yield the same end result, but the -1 approach seems to be a bit more unfair, since it would be purely for reasons we don't (yet) apply to currently-integrated projects... The one concern I have with a small core is that there is not an easy way to assess the maturity of a project on stackforge. The stackforge projects may be missing packaging, Red Hat testing, puppet modules, install/admin documentation etc. Thus, I need to have some indication that a project is deployable before looking at it with my user community to see if it meets a need that is sustainable. Do you see the optional layer services being blessed / validated in some way and therefore being easy to identify ? Tim -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On Thu, Sep 4, 2014 at 12:39 PM, Tim Bell tim.b...@cern.ch wrote: -Original Message- From: Thierry Carrez [mailto:thie...@openstack.org] Sent: 04 September 2014 16:59 To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting Sean Dague wrote: [...] So, honestly, I'll probably remain -1 on the final integration vote, not because Zaqar is bad, but because I'm feeling more firmly that for OpenStack to not leave the small deployers behind we need to redefine the tightly integrated piece of OpenStack to basically the Layer 1 2 parts of my diagram, and consider the rest of the layers exciting parts of our ecosystem that more advanced users may choose to deploy to meet their needs. Smaller tent, big ecosystem, easier on ramp. I realize that largely means Zaqar would be caught up in a definition discussion outside of it's control, and that's kind of unfortunate, as Flavio and team have been doing a bang up job of late. But we need to stop considering integration as the end game of all interesting software in the OpenStack ecosystem, and I think it's better to have that conversation sooner rather than later. I think it's pretty clear at this point that: (1) we need to have a discussion about layers (base nucleus, optional extra services at the very least) and the level of support we grant to each -- the current binary approach is not working very well (2) If we accept Zaqar next week, it's pretty clear it would not fall in the base nucleus layer but more in an optional extra services layer, together with at the very least Trove and Sahara There are two ways of doing this: follow Sean's approach and -1 integration (and have zaqar apply to that optional layer when we create it), or +1 integration now (and have zaqar follow whichever other integrated projects we place in that layer when we create it). I'm still hesitating on the best approach. I think they yield the same end result, but the -1 approach seems to be a bit more unfair, since it would be purely for reasons we don't (yet) apply to currently-integrated projects... The one concern I have with a small core is that there is not an easy way to assess the maturity of a project on stackforge. The stackforge projects may be missing packaging, Red Hat testing, puppet modules, install/admin documentation etc. Thus, I need to have some indication that a project is deployable before looking at it with my user community to see if it meets a need that is sustainable. I identify with this concern as Docs PTL. That said, it's very difficult already even if a project is under the openstack github org to know how well documented it is. Do you see the optional layer services being blessed / validated in some way and therefore being easy to identify ? I've already started to work on doc requirements for incubating and integrating projects [1], which may help with some checklist creation. My thinking is, if you can't document it, it's not well-validated. Is that reasonable, to adopt a docs or it didn't happen stance here? Thanks for thinking on this -- I do like the layers approach and I think we just need to define expectations for layers so all the community can do a quick sanity check. Anne 1. https://wiki.openstack.org/wiki/Documentation/IncubateIntegrate Tim -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
Thanks for your comments Gordon. I appreciate where you are coming from and I think we are actually in agreement on a lot of things. I just want to make it clear that from the very beginning of the project the team has tried to communicate (but perhaps could have done a better job at it) that we aren’t trying to displace other messaging systems that are clearly delivering a lot of value today. In fact, I personally have long been a proponent of using the best tool for the job. The Zaqar project was kicked off at an unconference session several summits ago because the community saw a need that was not covered by other messaging systems. Does that mean those other systems are “bad” or “wrong”? Of course not. It simply means that there are some cases where those other systems aren’t the best tool for the job, and another tool is needed (and vice versa). Does that other tool look *exactly* like Zaqar? Probably not. But a lot of people have told us Zaqar--in its current form--already delivers a lot of value that they can’t get from other messaging systems that are available to them. Zaqar, like any open source project, is a manifestation of lots of peoples' ideas, and will evolve over time to meet the needs of the community. Does a Qpid/Rabbit/Kafka provisioning service make sense? Probably. Would such a service totally overlap in terms of use-cases with Zaqar? Community feedback suggests otherwise. Will there be some other kind of thing that comes out of the woodwork? Possibly. (Heck, if something better comes along I for one have no qualms in shifting resources to the more elegant solution--again, use the best tool for the job.) This process happens all the time in the broader open-source world. But this process takes a healthy amount of time, plus broad exposure and usage, which is something that you simply don’t get as a non-integrated project in the OpenStack ecosystem. In any case, it’s pretty clear to me that Zaqar graduating should not be viewed as making it the officially blessed messaging service for the cloud” and nobody is allowed to have any other ideas, ever. If that happens, it’s only a symptom of a deeper perception/process problem that is far from unique to Zaqar. In fact, I think it touches on all non-integrated projects, and many integrated ones as well. --Kurt On 9/4/14, 12:38 PM, Gordon Sim g...@redhat.com wrote: This is not intended as criticism of Zaqar in anyway. In my opinion, 'reinvention' is not necessarily a bad thing. It's just another way of saying innovative approach and/or original thinking, both of which are good and both of which I think are important in the context of communicating in the cloud. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
