Re: [openstack-dev] [heat] Stack update and raw_template backup
On 30-Jul-14 23:24, Zane Bitter wrote: > On 30/07/14 02:21, Anant Patil wrote: >> On 28-Jul-14 22:37, Clint Byrum wrote: >>> Excerpts from Zane Bitter's message of 2014-07-28 07:25:24 -0700: On 26/07/14 00:04, Anant Patil wrote: > When the stack is updated, a diff of updated template and current > template can be stored to optimize database. And perhaps Heat should > have an API to retrieve this history of templates for inspection etc. > when the stack admin needs it. If there's a demand for that feature we could implement it, but it doesn't easily fall out of the current implementation any more. >>> >>> We are never going to do it even 1/10th as well as git. In fact we won't >>> even do it 1/0th as well as CVS. >>> >>> ___ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >> Zane, >> I am working the defect you had filed, which would clean up backup stack >> along with the resources, templates and other data. >> >> However, I simply don't want to delete the templates for the same reason >> as we don't hard-delete the stack. Anyone who deploys a stack and >> updates it over time would want to view the the updates in the templates >> for debugging or auditing reasons. > > As I mentioned, and as Ton mentioned in another thread, the old > templates are useless for auditing (and to a large extent debugging) > because the update process turns them into Frankentemplates that don't > reflect either the old or the new template supplied by the user (though > they're much closer to the new than the old). > > So if you don't delete them, then you don't end up with a copy of the > old template, you end up with a broken copy of the new template. > >> It is not fair to assume that every >> user has a VCS with him to store the templates. > > It most certainly is fair to assume that. > > In addition, Glance has an artifact repository project already underway > with the goal of providing versioned access to templates as part of > OpenStack. > > It's likely that not all users are making use of a VCS, but if they're > not then I don't know why they bother using Heat. The whole point of the > project is to provide a way for people to describe their infrastructure > in a way that _can_ be managed in a VCS. Whenever we add new features, > we always try to do so in a way that _encourages_ users to store their > templates in a VCS and _discourages_ them from managing them in an ad > hoc manner. > >> It is kind of >> inconvenience for me to not have the ability to view my updates in >> templates. > > I tend to agree that it would be kind-of nice to have, but you're > talking about it as if it's a simple matter of just not deleting the old > template and sticking an API in front of it, rather than the major new > development that it actually is. > >> We need not go as far as git or any VCS. Any library which can do a diff >> and patch of text files can be used, like the google-diff-match-patch. > > We don't store the original text of templates - in heat-engine we only > get the object tree obtained by parsing the JSON or YAML. So the > templates we store or could store currently are of not much use to > (human) users. > > cheers, > Zane. > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > Sorry for coming late on this. Thanks Zane for clarifications. However, > I tend to agree that it would be kind-of nice to have, but you're > talking about it as if it's a simple matter of just not deleting the > old template and sticking an API in front of it, rather than the major > new development that it actually is. I am not really saying that it would be that easy. The how part will come later. I am discussing this in the context of bug I am fixing but certainly it's not that simple. The templates, resources, events etc are all integral part of stack. I would like to just grab the heat command to view all the stack and its updates and templates. May be I should try to put it in a blueprint so that the idea is clear and we can discuss. - Anant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] Stack update and raw_template backup
On 30/07/14 02:21, Anant Patil wrote: On 28-Jul-14 22:37, Clint Byrum wrote: Excerpts from Zane Bitter's message of 2014-07-28 07:25:24 -0700: On 26/07/14 00:04, Anant Patil wrote: When the stack is updated, a diff of updated template and current template can be stored to optimize database. And perhaps Heat should have an API to retrieve this history of templates for inspection etc. when the stack admin needs it. If there's a demand for that feature we could implement it, but it doesn't easily fall out of the current implementation any more. We are never going to do it even 1/10th as well as git. In fact we won't even do it 1/0th as well as CVS. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Zane, I am working the defect you had filed, which would clean up backup stack along with the resources, templates and other data. However, I simply don't want to delete the templates for the same reason as we don't hard-delete the stack. Anyone who deploys a stack and updates it over time would want to view the the updates in the templates for debugging or auditing reasons. As I mentioned, and as Ton mentioned in another thread, the old templates are useless for auditing (and to a large extent debugging) because the update process turns them into Frankentemplates that don't reflect either the old or the new template supplied by the user (though they're much closer to the new than the old). So if you don't delete them, then you don't end up with a copy of the old template, you end up with a broken copy of the new template. It is not fair to assume that every user has a VCS with him to store the templates. It most certainly is fair to assume that. In addition, Glance has an artifact repository project already underway with the goal of providing versioned access to templates as part of OpenStack. It's likely that not all users are making use of a VCS, but if they're not then I don't know why they bother using Heat. The whole point of the project is to provide a way for people to describe their infrastructure in a way that _can_ be managed in a VCS. Whenever we add new features, we always try to do so in a way that _encourages_ users to store their templates in a VCS and _discourages_ them from managing them in an ad hoc manner. It is kind of inconvenience for me to not have the ability to view my updates in templates. I tend to agree that it would be kind-of nice to have, but you're talking about it as if it's a simple matter of just not deleting the old template and sticking an API in front of it, rather than the major new development that it actually is. We need not go as far as git or any VCS. Any library which can do a diff and patch of text files can be used, like the google-diff-match-patch. We don't store the original text of templates - in heat-engine we only get the object tree obtained by parsing the JSON or YAML. So the templates we store or could store currently are of not much use to (human) users. cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] Stack update and raw_template backup
On 28-Jul-14 22:37, Clint Byrum wrote: > Excerpts from Zane Bitter's message of 2014-07-28 07:25:24 -0700: >> On 26/07/14 00:04, Anant Patil wrote: >>> When the stack is updated, a diff of updated template and current >>> template can be stored to optimize database. And perhaps Heat should >>> have an API to retrieve this history of templates for inspection etc. >>> when the stack admin needs it. >> >> If there's a demand for that feature we could implement it, but it >> doesn't easily fall out of the current implementation any more. > > We are never going to do it even 1/10th as well as git. In fact we won't > even do it 1/0th as well as CVS. > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > Zane, I am working the defect you had filed, which would clean up backup stack along with the resources, templates and other data. However, I simply don't want to delete the templates for the same reason as we don't hard-delete the stack. Anyone who deploys a stack and updates it over time would want to view the the updates in the templates for debugging or auditing reasons. It is not fair to assume that every user has a VCS with him to store the templates. It is kind of inconvenience for me to not have the ability to view my updates in templates. We need not go as far as git or any VCS. Any library which can do a diff and patch of text files can be used, like the google-diff-match-patch. - Anant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] Stack update and raw_template backup
Excerpts from Zane Bitter's message of 2014-07-28 07:25:24 -0700: > On 26/07/14 00:04, Anant Patil wrote: > > When the stack is updated, a diff of updated template and current > > template can be stored to optimize database. And perhaps Heat should > > have an API to retrieve this history of templates for inspection etc. > > when the stack admin needs it. > > If there's a demand for that feature we could implement it, but it > doesn't easily fall out of the current implementation any more. We are never going to do it even 1/10th as well as git. In fact we won't even do it 1/0th as well as CVS. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] Stack update and raw_template backup
On 26/07/14 00:04, Anant Patil wrote: Hi, When we do a stack update, I see that there are 2 copies of raw_template stored in database for each update. For n updates there are "2n + 1" entries of raw_template in database. Is this expected or is it a bug? When I dug more into it, I see that the deep copy of template is not copying the template.id resulting in it being stored when we do backup_stack.store and new_stack.store. It's expected that we store 2 new templates, but the fact that we are not deleting the backup stack's template and the stack's old template after a successful update is a bug. Ideally, we should be keeping one copy of old template when the stack is updated. With this we have the history of templates updated over time. With the changes that I made a few weeks ago, we now modify the existing template during an update, so we wouldn't really have the history anyway. When the stack is updated, a diff of updated template and current template can be stored to optimize database. And perhaps Heat should have an API to retrieve this history of templates for inspection etc. when the stack admin needs it. If there's a demand for that feature we could implement it, but it doesn't easily fall out of the current implementation any more. cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [heat] Stack update and raw_template backup
Hi, When we do a stack update, I see that there are 2 copies of raw_template stored in database for each update. For n updates there are "2n + 1" entries of raw_template in database. Is this expected or is it a bug? When I dug more into it, I see that the deep copy of template is not copying the template.id resulting in it being stored when we do backup_stack.store and new_stack.store. Ideally, we should be keeping one copy of old template when the stack is updated. With this we have the history of templates updated over time. When the stack is updated, a diff of updated template and current template can be stored to optimize database. And perhaps Heat should have an API to retrieve this history of templates for inspection etc. when the stack admin needs it. Please share your thoughts! - Anant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
