On 12/23/2015 04:19 AM, Dougal Matthews wrote:
On 22 December 2015 at 17:59, Ben Nemec <[email protected]
<mailto:[email protected]>> wrote:
Can we just do git like I've been suggesting all along? ;-)
More serious discussion inline. :-)
On 12/22/2015 09:36 AM, Dougal Matthews wrote:
> Hi all,
>
> This topic came up in the 2015-12-15 meeting[1], and again
briefly today.
> After working with the code that came out of the deployment library
> spec[2] I
> had some concerns with how we are storing the templates.
>
> Simply put, when we are dealing with 100+ files from
tripleo-heat-templates
> how can we ensure consistency in Swift without any atomicity or
> transactions.
> I think this is best explained with a couple of examples.
>
> - When we create a new deployment plan (upload all the templates
to swift)
> how do we handle the case where there is an error? For
example, if we are
> uploading 10 files - what do we do if the 5th one fails for
some reason?
> There is a patch to do a manual rollback[3], but I have
concerns about
> doing this in Python. If Swift is completely inaccessible for
a short
> period the rollback wont work either.
>
> - When deploying to Heat, we need to download all the YAML files
from
> Swift.
> This can take a couple of seconds. What happens if somebody
starts to
> upload a new version of the plan in the middle? We could end
up trying to
> deploy half old and half new files. We wouldn't have a
consistent view of
> the database.
>
> We had a few suggestions in the meeting:
>
> - Add a locking mechanism. I would be concerned about deadlocks or
> having to
> lock for the full duration of a deploy.
There should be no need to lock the plan for the entire deploy. It's
not like we're re-reading the templates at the end of the deploy today.
It's a one-shot read and then the plan could be unlocked, at least as
far as I know.
Good point. That would be holding the lock for longer than we need.
The only option where we wouldn't need locking at all is the
read-copy-update model Clint mentions, which might be a valid option as
well. Whatever we do, there are going to be concurrency issues though.
For example, what happens if two users try to make updates to the plan
at the same time? If you don't either merge the changes or disallow one
of them completely then one user's changes might be lost.
TBH, this is further convincing me that we should just make this git
backed and let git handle the merging and conflict resolution (never
mind the fact that it gets us a well-understood version control system
for "free"). For updates that don't conflict with other changes, git
can merge them automatically, but for merge conflicts you just return a
rebase error to the user and make them resolve it. I have a feeling
this is the behavior we'll converge on eventually anyway, and rather
than reimplement git, let's just use the real thing.
I'd be curious to hear more how you would go about doing this with git. I've
never automated git to this level, so I am concerned about what issues we
might hit.
Do you happen to know of any good sources that I can find out more?
One source of inspiration could be the way Openshift handles things. You
get a git URL to push your $whatever to, and they have git hooks that
deploy when you push. This would be perfect for merge conflict
situations. Imagine two cases.
Normal case:
1. User does stuff in web interface or over API
2. The API adds templates and such to git repo, then deploys as needed
3. User is happy!
Merge conflict case:
1. User does stuff in web interface or over API
2. The API tries, but there's a merge conflict. It hands back a 409 and
a git URL
3. User clones the repo and resolves conflict
4. User pushes repo back up
5. API deploys stuff
6. User is happy!
/2 cents
> - Store the files in a tarball (so we only deal with one file). I think
we
> could still hit issues with multiple operations unless we guarantee
that
> only one instance of the API is ever running.
>
> I think these could potentially be made to work, but at this point are we
> using the best tool for the job?
>
> For alternatives, I see a can think of a couple of options:
>
> - Use a database, like we did for Tuskar and most OpenStack API's do.
I kind of like this, in particular because it would allow us to handle
migrations between versions the same way as other OpenStack services.
Yeah, I feel like this is the "safe" option. It's a well trodden and
tested path.
I'm not entirely sure how it maps to our template configuration method
though. Storing a bunch of template blobs in the database feels a
little square peg, round hole to me, and might undo a lot of the
benefits of using the database in the first place.
I don't follow this point. In the way we access Swift, everything is
essentially
a blob of text also.
Now, on the other hand, having a database to store basic data like
metadata on plans and such might be a useful thing regardless of where
we store the templates themselves. We could also use it for locking
just fine IMHO - TripleO isn't a tool targeted to have cloud-scale
number of users. It's a deployer tool with a relatively limited number
of users per installation, so the scaling of a locking mechanism isn't a
problem I necessarily think we need to solve. We have way bigger
scaling issues than that to tackle before I think we would hit the limit
of a database-locking scheme.
> - Invest time in building something on Swift.
> - Glance was transitioning to be a Artifact store. I don't know the
> status of
> this or if it would meet out needs.
>
> Any input, ideas or suggestions would be great!
>
> Thanks,
> Dougal
>
>
> [1]:
>http://eavesdrop.openstack.org/meetings/tripleo/2015/tripleo.2015-12-15-14.03.log.html#l-89
> [2]:
>https://specs.openstack.org/openstack/tripleo-specs/specs/mitaka/tripleo-overcloud-deployment-library.html
> [3]:https://review.openstack.org/#/c/257481/
>
>
>
__________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
[email protected]?subject:unsubscribe
<http://[email protected]?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
[email protected]?subject:unsubscribe
<http://[email protected]?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Ryan Brown / Senior Software Engineer, Openstack / Red Hat, Inc.
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
