Re: [openstack-dev] Moving task flow to conductor - concern about scale

2013-07-20 Thread Joshua Harlow
Looking at the conductor code it still to me provides a low level database API that succumbs to the same races as a the old db access did. Get calls followed by some response followed by some python code followed by some rpc update followed by more code is still susceptible to consistency

Re: [openstack-dev] Moving task flow to conductor - concern about scale

2013-07-19 Thread Day, Phil
-Original Message- From: Dan Smith [mailto:d...@danplanet.com] Sent: 16 July 2013 14:51 To: OpenStack Development Mailing List Cc: Day, Phil Subject: Re: [openstack-dev] Moving task flow to conductor - concern about scale In the original context of using Conductor as a database

Re: [openstack-dev] Moving task flow to conductor - concern about scale

2013-07-19 Thread Day, Phil
Mailing List Cc: OpenStack Development Mailing List Subject: Re: [openstack-dev] Moving task flow to conductor - concern about scale Hi Phil, I understand and appreciate your concern and I think everyone is trying to keep that in mind. It still appears to me to be to early

Re: [openstack-dev] Moving task flow to conductor - concern about scale

2013-07-19 Thread Dan Smith
There's nothing I've seen so far that causes me alarm, but then again we're in the very early stages and haven't moved anything really complex. The migrations (live, cold, and resize) are moving there now. These are some of the more complex stateful operations I would expect conductor to

Re: [openstack-dev] Moving task flow to conductor - concern about scale

2013-07-19 Thread Peter Feiner
On Fri, Jul 19, 2013 at 11:06 AM, Dan Smith d...@danplanet.com wrote: FWIW, I don't think anyone is suggesting a single conductor, and especially not a single database proxy. This is a critical detail that I missed. Re-reading Phil's original email, I see you're debating the ratio of

Re: [openstack-dev] Moving task flow to conductor - concern about scale

2013-07-19 Thread Dan Smith
I had assumed that some of the task management state would exist in memory. Is it all going to exist in the database? Well, our state is tracked in the database now, so.. yeah. There's a desire, of course, to make the state transitions as idempotent/restartable as possible, which may mean

Re: [openstack-dev] Moving task flow to conductor - concern about scale

2013-07-19 Thread Joe Gordon
On Jul 19, 2013 9:57 AM, Day, Phil philip@hp.com wrote: -Original Message- From: Dan Smith [mailto:d...@danplanet.com] Sent: 19 July 2013 15:15 To: OpenStack Development Mailing List Cc: Day, Phil Subject: Re: [openstack-dev] Moving task flow to conductor - concern about

Re: [openstack-dev] Moving task flow to conductor - concern about scale

2013-07-19 Thread Day, Phil
-Original Message- From: Dan Smith [mailto:d...@danplanet.com] Sent: 19 July 2013 15:15 To: OpenStack Development Mailing List Cc: Day, Phil Subject: Re: [openstack-dev] Moving task flow to conductor - concern about scale There's nothing I've seen so far that causes me alarm

Re: [openstack-dev] Moving task flow to conductor - concern about scale

2013-07-19 Thread Peter Feiner
On Fri, Jul 19, 2013 at 10:15 AM, Dan Smith d...@danplanet.com wrote: So rather than asking what doesn't work / might not work in the future I think the question should be aside from them both being things that could be described as a conductor - what's the architectural reason for

Re: [openstack-dev] Moving task flow to conductor - concern about scale

2013-07-19 Thread Dan Smith
Nova-conductor is the gateway to the database for nova-compute processes. So permitting a single nova-conductor process would effectively serialize all database queries during instance creation, deletion, periodic instance refreshes, etc. FWIW, I don't think anyone is suggesting a single

Re: [openstack-dev] Moving task flow to conductor - concern about scale

2013-07-19 Thread Robert Collins
On 19 July 2013 22:55, Day, Phil philip@hp.com wrote: Hi Josh, My idea's really pretty simple - make DB proxy and Task workflow separate services, and allow people to co-locate them if they want to. +1, for all the reasons discussed in this thread. I was weirded out when I saw

Re: [openstack-dev] Moving task flow to conductor - concern about scale

2013-07-17 Thread Joshua Harlow
Hi Phil, I understand and appreciate your concern and I think everyone is trying to keep that in mind. It still appears to me to be to early in this refactoring and task restructuring effort to tell where it may end up. I think that's also good news since we can get these kinds of ideas

[openstack-dev] Moving task flow to conductor - concern about scale

2013-07-16 Thread Day, Phil
Hi Folks, Reviewing some the changes to move control flows into conductor made me wonder about an issue that I haven't seen discussed so far (apologies if it was and I've missed it): In the original context of using Conductor as a database proxy then the number of conductor instances is

Re: [openstack-dev] Moving task flow to conductor - concern about scale

2013-07-16 Thread Dan Smith
In the original context of using Conductor as a database proxy then the number of conductor instances is directly related to the number of compute hosts I need them to serve. Just a point of note, as far as I know, the plan has always been to establish conductor as a thing that sits between