[Openstack] Multi-Cluster/Zone - Devil in the Details ...

2011-02-16 Thread Sandy Walsh
Hi y'all Like they say The devil is in the details. I'm at the stage where the parent zones will talk to the child zones and there are some interesting implementation issues: Problem 1. I'd like to pass the incoming HTTP Request object along to the Scheduler so I don't have to remarshall the

Re: [Openstack] Multi-Cluster/Zone - Devil in the Details ...

2011-02-16 Thread Ed Leafe
On Feb 16, 2011, at 10:26 AM, Sandy Walsh wrote: But this introduces a problem. Consider this use-case: a. I issue a create-instance via the top-level API in zone-A b. the request is relayed down to zone-C c. the instance is created some time later Q1. How does the user learn what the

Re: [Openstack] Multi-Cluster/Zone - Devil in the Details ...

2011-02-16 Thread Sandy Walsh
Thanks for feedback Ed. Comments in [] below ... From: Ed Leafe [e...@leafe.com] On Feb 16, 2011, at 10:26 AM, Sandy Walsh wrote: Isn't the instance name usually supplied by the user/originator? [Sorry, yes the instance name is passed in on the

Re: [Openstack] Multi-Cluster/Zone - Devil in the Details ...

2011-02-16 Thread Ed Leafe
On Feb 16, 2011, at 11:54 AM, Sandy Walsh wrote: Thanks for feedback Ed. Comments in [] below ... Yeesh - that makes for ugly quoting. :) Let me try to reformat the quoting. Isn't the instance name usually supplied by the user/originator? [Sorry, yes the instance name is

Re: [Openstack] Multi-Cluster/Zone - Devil in the Details ...

2011-02-16 Thread Jay Pipes
On Wed, Feb 16, 2011 at 12:16 PM, Ed Leafe e...@leafe.com wrote: On Feb 16, 2011, at 11:54 AM, Sandy Walsh wrote:        Isn't the instance name usually supplied by the user/originator? [Sorry, yes the instance name is passed in on the request, but the instance ID is what's needed (assuming

Re: [Openstack] Multi-Cluster/Zone - Devil in the Details ...

2011-02-16 Thread Monsyne Dragon
On 2/16/11 9:26 AM, Sandy Walsh wrote: Hi y'all Like they say The devil is in the details. I'm at the stage where the parent zones will talk to the child zones and there are some interesting implementation issues: *Problem 1.* I'd like to pass the incoming HTTP Request object along to the

Re: [Openstack] Multi-Cluster/Zone - Devil in the Details ...

2011-02-16 Thread Sandy Walsh
Thanks Dragon! Had another conversation with Jay on this as well. I think the caching with the smart use of instance/customer naming will go a long way to helping the search. More comments below ... -S From: openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net

Re: [Openstack] Multi-Cluster/Zone - Devil in the Details ...

2011-02-16 Thread Eric Day
Hi Sandy, On Wed, Feb 16, 2011 at 06:19:52PM +, Sandy Walsh wrote: Hmm. You wouldn't really need to re-marshall the request. Just copy the needed headers url, and pass along the body as you received it. Basically you are just acting as a sort of http proxy.

Re: [Openstack] Multi-Cluster/Zone - Devil in the Details ...

2011-02-16 Thread Eric Day
On Wed, Feb 16, 2011 at 01:02:22PM -0500, Jay Pipes wrote: [Sorry, yes the instance name is passed in on the request, but the instance ID is what's needed (assuming of course instance ID is unique across zones.)]        The ID is determined early in the process; well before the request

Re: [Openstack] Multi-Cluster/Zone - Devil in the Details ...

2011-02-16 Thread Jay Pipes
On Wed, Feb 16, 2011 at 4:25 PM, Eric Day e...@oddments.org wrote: On Wed, Feb 16, 2011 at 03:59:10PM -0500, Jay Pipes wrote: But, as I mentioned to Sandy on IRC, caching and performance should be a secondary concern. The primary concern, right now, is just making this all work. In other

Re: [Openstack] Multi-Cluster/Zone - Devil in the Details ...

2011-02-16 Thread Eric Day
On Wed, Feb 16, 2011 at 04:33:06PM -0500, Jay Pipes wrote: While I would agree with this most of the time, there are some cases where optimizing later just doesn't work. Or, optimizing means rewriting everything you've done and replacing it with something that will scale seamlessly. I've

Re: [Openstack] Multi-Cluster/Zone - Devil in the Details ...

2011-02-16 Thread Soren Hansen
2011/2/16 Ed Leafe e...@leafe.com: This was one of the issues we discussed during the sprint planning. I believe (check with cyn) that the consensus was to use a caching strategy akin to DNS: e.g., if zone A got a request for instance ID=12345, it would check to see if it had id 12345 in

Re: [Openstack] Multi-Cluster/Zone - Devil in the Details ...

2011-02-16 Thread Eric Day
Good points Soren, and this is why I was suggesting we not solve this problem with a cache, but instead an eventually consistent replication stream of the aggregate data. -Eric On Wed, Feb 16, 2011 at 11:12:50PM +0100, Soren Hansen wrote: 2011/2/16 Ed Leafe e...@leafe.com: This was one of the

Re: [Openstack] Multi-Cluster/Zone - Devil in the Details ...

2011-02-16 Thread Soren Hansen
2011/2/16 Eric Day e...@oddments.org: This doesn't help the 'list all instances' search. This would be very expensive when dealing with a large number of accounts and instances. We need a more active caching policy, which ends up being more of a replication subset than a cache. Did we ever

Re: [Openstack] Multi-Cluster/Zone - Devil in the Details ...

2011-02-16 Thread Ed Leafe
On Feb 16, 2011, at 5:12 PM, Soren Hansen wrote: * Whenever I ask something for information and I get out-of-date, cached data back I feel like I'm back in 2003. And 2003 sucked, I might add. The other side of that coin was the implementation of pub/sub. New/deleted/changed instances