Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On 2014-03-15 12:23, Solly Ross wrote: --nodeps will only sync the modules specified on the command line: https://wiki.openstack.org/wiki/Oslo#Syncing_Code_from_Incubator Heh, whoops. Must have missed that. Is it in the README/info at the top of the update.py script? It wasn't, but I pushed a change to add it: https://review.openstack.org/#/c/81004/ That said, it's not always safe to do that. You might sync a change in one module that depends on a change in another module and end up breaking something. It might not be caught in the sync either because the Oslo unit tests don't get synced across. Hmm... I suppose this is why we have libraries with dependencies (not meant to sound snarky). Although in the case of updating a library that you wrote, it's less likely to break things. Yeah, ideally you would never need nodeps because the deps would already be up to date when you do your sync, but unfortunately we aren't that good about keeping Oslo syncs current right now. Hopefully that will get better in Juno, but we'll see. Best Regards, Solly Ross - Original Message - From: "Ben Nemec" To: "OpenStack Development Mailing List (not for usage questions)" Cc: "Solly Ross" Sent: Friday, March 14, 2014 4:36:03 PM Subject: Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py On 2014-03-14 14:49, Solly Ross wrote: It would also be great if there was a way to only sync one package. There is. :-) --nodeps will only sync the modules specified on the command line: https://wiki.openstack.org/wiki/Oslo#Syncing_Code_from_Incubator That said, it's not always safe to do that. You might sync a change in one module that depends on a change in another module and end up breaking something. It might not be caught in the sync either because the Oslo unit tests don't get synced across. When adding a new library to a project (e.g. openstack.common.report to Nova), one would want to only sync the openstack.common.report parts, and not the any changes from the rest of openstack.common. My process has been 1. Edit openstack-common.conf to only contain the packages I want 2. Run the update 3. Make sure there wasn't code that didn't get changed from 'openstack.common.xyz' to 'nova.openstack.common.xyz' (hint: this happens some times) 4. git checkout openstack-common.conf to revert the changes to openstack-common.conf IMHO, update.py needs a bit of work (well, I think the whole code copying thing needs a bit of work, but that's a different story). Best Regards, Solly Ross - Original Message - From: "Jay S Bryant" To: "OpenStack Development Mailing List (not for usage questions)" Sent: Friday, March 14, 2014 3:36:49 PM Subject: Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py From: Brant Knudson To: "OpenStack Development Mailing List (not for usage questions)" , Date: 03/14/2014 02:21 PM Subject: Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py On Fri, Mar 14, 2014 at 2:05 PM, Jay S Bryant < jsbry...@us.ibm.com > wrote: It would be great if we could get the process for this automated. In the mean time, those of us doing the syncs will just have to slog through the process. Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev What's the process? How do I generate the list of changes? Brant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Brant, My process thus far has been the following: 1. Do the sync to see what files are changed. 2. Take a look at the last commit sync'd to what is currently in master for a file. 3. Document all the commits that have come in on that file since. 4. Repeat process for all the relevant files if there is more than one. 5. If are multiples files I organize the commits with a list of the files touched by that commit. 6. Document the master level of Oslo when the sync was done for reference. Process may not be perfect, but it gets the job done. Here is an example of the format I use: https://review.openstack.org/#/c/75740/ Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
> --nodeps will only sync the modules specified on the command line: > https://wiki.openstack.org/wiki/Oslo#Syncing_Code_from_Incubator Heh, whoops. Must have missed that. Is it in the README/info at the top of the update.py script? > That said, it's not always safe to do that. You might sync a change in > one module that depends on a change in another module and end up > breaking something. It might not be caught in the sync either because > the Oslo unit tests don't get synced across. Hmm... I suppose this is why we have libraries with dependencies (not meant to sound snarky). Although in the case of updating a library that you wrote, it's less likely to break things. Best Regards, Solly Ross - Original Message - From: "Ben Nemec" To: "OpenStack Development Mailing List (not for usage questions)" Cc: "Solly Ross" Sent: Friday, March 14, 2014 4:36:03 PM Subject: Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py On 2014-03-14 14:49, Solly Ross wrote: > It would also be great if there was a way to only sync one package. There is. :-) --nodeps will only sync the modules specified on the command line: https://wiki.openstack.org/wiki/Oslo#Syncing_Code_from_Incubator That said, it's not always safe to do that. You might sync a change in one module that depends on a change in another module and end up breaking something. It might not be caught in the sync either because the Oslo unit tests don't get synced across. > When adding a new library > to a project (e.g. openstack.common.report to Nova), one would want to > only sync the openstack.common.report > parts, and not the any changes from the rest of openstack.common. My > process has been > > 1. Edit openstack-common.conf to only contain the packages I want > 2. Run the update > 3. Make sure there wasn't code that didn't get changed from > 'openstack.common.xyz' to 'nova.openstack.common.xyz' (hint: this > happens some times) > 4. git checkout openstack-common.conf to revert the changes to > openstack-common.conf > > IMHO, update.py needs a bit of work (well, I think the whole code > copying thing needs a bit of work, but that's a different story). > > Best Regards, > Solly Ross > > - Original Message ----- > From: "Jay S Bryant" > To: "OpenStack Development Mailing List (not for usage questions)" > > Sent: Friday, March 14, 2014 3:36:49 PM > Subject: Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py > > > > > From: Brant Knudson > To: "OpenStack Development Mailing List (not for usage questions)" > , > Date: 03/14/2014 02:21 PM > Subject: Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py > > > > > > > > On Fri, Mar 14, 2014 at 2:05 PM, Jay S Bryant < jsbry...@us.ibm.com > > wrote: > > It would be great if we could get the process for this automated. In > the > mean time, those of us doing the syncs will just have to slog through > the > process. > > Jay > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > What's the process? How do I generate the list of changes? > > Brant > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > Brant, > > My process thus far has been the following: > > > 1. Do the sync to see what files are changed. > 2. Take a look at the last commit sync'd to what is currently in > master for a file. > 3. Document all the commits that have come in on that file since. > 4. Repeat process for all the relevant files if there is more than > one. > 5. If are multiples files I organize the commits with a list of > the files touched by that commit. > 6. Document the master level of Oslo when the sync was done for > reference. > > Process may not be perfect, but it gets the job done. Here is an > example of the format I use: https://review.openstack.org/#/c/75740/ > > Jay > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On 2014-03-14 14:49, Solly Ross wrote: It would also be great if there was a way to only sync one package. There is. :-) --nodeps will only sync the modules specified on the command line: https://wiki.openstack.org/wiki/Oslo#Syncing_Code_from_Incubator That said, it's not always safe to do that. You might sync a change in one module that depends on a change in another module and end up breaking something. It might not be caught in the sync either because the Oslo unit tests don't get synced across. When adding a new library to a project (e.g. openstack.common.report to Nova), one would want to only sync the openstack.common.report parts, and not the any changes from the rest of openstack.common. My process has been 1. Edit openstack-common.conf to only contain the packages I want 2. Run the update 3. Make sure there wasn't code that didn't get changed from 'openstack.common.xyz' to 'nova.openstack.common.xyz' (hint: this happens some times) 4. git checkout openstack-common.conf to revert the changes to openstack-common.conf IMHO, update.py needs a bit of work (well, I think the whole code copying thing needs a bit of work, but that's a different story). Best Regards, Solly Ross - Original Message - From: "Jay S Bryant" To: "OpenStack Development Mailing List (not for usage questions)" Sent: Friday, March 14, 2014 3:36:49 PM Subject: Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py From: Brant Knudson To: "OpenStack Development Mailing List (not for usage questions)" , Date: 03/14/2014 02:21 PM Subject: Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py On Fri, Mar 14, 2014 at 2:05 PM, Jay S Bryant < jsbry...@us.ibm.com > wrote: It would be great if we could get the process for this automated. In the mean time, those of us doing the syncs will just have to slog through the process. Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev What's the process? How do I generate the list of changes? Brant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Brant, My process thus far has been the following: 1. Do the sync to see what files are changed. 2. Take a look at the last commit sync'd to what is currently in master for a file. 3. Document all the commits that have come in on that file since. 4. Repeat process for all the relevant files if there is more than one. 5. If are multiples files I organize the commits with a list of the files touched by that commit. 6. Document the master level of Oslo when the sync was done for reference. Process may not be perfect, but it gets the job done. Here is an example of the format I use: https://review.openstack.org/#/c/75740/ Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
It would also be great if there was a way to only sync one package. When adding a new library to a project (e.g. openstack.common.report to Nova), one would want to only sync the openstack.common.report parts, and not the any changes from the rest of openstack.common. My process has been 1. Edit openstack-common.conf to only contain the packages I want 2. Run the update 3. Make sure there wasn't code that didn't get changed from 'openstack.common.xyz' to 'nova.openstack.common.xyz' (hint: this happens some times) 4. git checkout openstack-common.conf to revert the changes to openstack-common.conf IMHO, update.py needs a bit of work (well, I think the whole code copying thing needs a bit of work, but that's a different story). Best Regards, Solly Ross - Original Message - From: "Jay S Bryant" To: "OpenStack Development Mailing List (not for usage questions)" Sent: Friday, March 14, 2014 3:36:49 PM Subject: Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py From: Brant Knudson To: "OpenStack Development Mailing List (not for usage questions)" , Date: 03/14/2014 02:21 PM Subject: Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py On Fri, Mar 14, 2014 at 2:05 PM, Jay S Bryant < jsbry...@us.ibm.com > wrote: It would be great if we could get the process for this automated. In the mean time, those of us doing the syncs will just have to slog through the process. Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev What's the process? How do I generate the list of changes? Brant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Brant, My process thus far has been the following: 1. Do the sync to see what files are changed. 2. Take a look at the last commit sync'd to what is currently in master for a file. 3. Document all the commits that have come in on that file since. 4. Repeat process for all the relevant files if there is more than one. 5. If are multiples files I organize the commits with a list of the files touched by that commit. 6. Document the master level of Oslo when the sync was done for reference. Process may not be perfect, but it gets the job done. Here is an example of the format I use: https://review.openstack.org/#/c/75740/ Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
From: Brant Knudson To: "OpenStack Development Mailing List (not for usage questions)" , Date: 03/14/2014 02:21 PM Subject: Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py On Fri, Mar 14, 2014 at 2:05 PM, Jay S Bryant wrote: It would be great if we could get the process for this automated. In the mean time, those of us doing the syncs will just have to slog through the process. Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev What's the process? How do I generate the list of changes? Brant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Brant, My process thus far has been the following: Do the sync to see what files are changed. Take a look at the last commit sync'd to what is currently in master for a file. Document all the commits that have come in on that file since. Repeat process for all the relevant files if there is more than one. If are multiples files I organize the commits with a list of the files touched by that commit. Document the master level of Oslo when the sync was done for reference. Process may not be perfect, but it gets the job done. Here is an example of the format I use: https://review.openstack.org/#/c/75740/ Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On Fri, Mar 14, 2014 at 2:05 PM, Jay S Bryant wrote: > > It would be great if we could get the process for this automated. In the > mean time, those of us doing the syncs will just have to slog through the > process. > > Jay > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > What's the process? How do I generate the list of changes? Brant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
From: Duncan Thomas To: "OpenStack Development Mailing List (not for usage questions)" , Date: 03/14/2014 11:49 AM Subject: Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py On 14 March 2014 16:10, Jay S Bryant wrote: > --> Duncan, It is important to know what commits are being brought over to > help provide a pointer to > --> the possible cause of subsequent bugs that arise. I.E. if we sync up > the DB, there is a commit for fixing > --> db connection order and suddenly we are getting intermittent DB > connection failures, it give us > --> a starting point to fixing the issue. Jay, there's been a mix-up in who's saying what here. I *very much* want to know what commits are being bought over. For slightly different reasons (I'm mostly wanting them for easy review, you for bug fixing). Brant is suggesting that just the last commit ID is enough, which I disagree with (and will continue to hit -1/-2 for). If somebody was to improve the import script to do this automatically that would be great. Currently I can't see an easy way of programatically telling when the last import was - I'll take another look at the problem if somebody smarter than me doesn't sort it first ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Duncan, Thanks for clarifying! I was quite confused as I thought you were in favor of the details in the commit messages. Sorry for confusing who was carrying what stance forward. Brant, I agree with Duncan. I think that having details of the commits that are being pulled in with a sync commit is important. I realize that the user probably isn't going to do a perfect job, but it certainly helps to give context to the changes being made and provides important bread crumbs for debug. It would be great if we could get the process for this automated. In the mean time, those of us doing the syncs will just have to slog through the process. Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On 14 March 2014 16:10, Jay S Bryant wrote: > --> Duncan, It is important to know what commits are being brought over to > help provide a pointer to > --> the possible cause of subsequent bugs that arise. I.E. if we sync up > the DB, there is a commit for fixing > --> db connection order and suddenly we are getting intermittent DB > connection failures, it give us > --> a starting point to fixing the issue. Jay, there's been a mix-up in who's saying what here. I *very much* want to know what commits are being bought over. For slightly different reasons (I'm mostly wanting them for easy review, you for bug fixing). Brant is suggesting that just the last commit ID is enough, which I disagree with (and will continue to hit -1/-2 for). If somebody was to improve the import script to do this automatically that would be great. Currently I can't see an easy way of programatically telling when the last import was - I'll take another look at the problem if somebody smarter than me doesn't sort it first ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
From: Brant Knudson To: "OpenStack Development Mailing List (not for usage questions)" , Date: 03/14/2014 10:26 AM Subject: Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py On Wed, Mar 12, 2014 at 11:03 AM, Duncan Thomas wrote: On 15 January 2014 18:53, Brant Knudson wrote: > At no point do I care what are the different commits that are being brought > in from oslo-incubator. If the commits are listed in the commit message then > I feel an obligation to verify that they got the right commits in the > message and that takes extra time for no gain. --> Duncan, It is important to know what commits are being brought over to help provide a pointer to --> the possible cause of subsequent bugs that arise. I.E. if we sync up the DB, there is a commit for fixing --> db connection order and suddenly we are getting intermittent DB connection failures, it give us --> a starting point to fixing the issue. I find that I very much *do* want a list of what changes have been pulled in, so I've so idea of the intent of the changes. Some of the OSLO changes can be large and complicated, and the more clues as to why things changed, the better the chance I've got of spotting breakages or differing assumptions between cinder and OSLO (of which there have been a number) I don't very often verify that the version that has been pulled in is the very latest or anything like that - generally I want to know: One thing that I think we should be verifying is that the changes being brought over have actually been committed to oslo-incubator. I'm sure there have been times where someone eager to get the fix in has not waited for the oslo-incubator merge before syncing their change over. - What issue are you trying to fix by doing an update? (The fact OSLO is ahead of us is rarely a good enough reason on its own to do an update - preferably reference a specific bug that exists in cinder) When I sync a change from oslo-incubator to fix a bug I put Closes-Bug on the commit message to indicate what bug is being fixed. If the sync tool was enhanced to pick out the *-Bug references from the oslo commits to include in the sync commit message that would be handy. - What other incidental changes are being pulled in (by intent, not just the code) - If I'm unsure about one of the incidental changes, how do I go find the history for it, with lots of searching (hense the commit ID or the change ID) - this lets me find bugs, reviews etc How does one get the list of commits that are being brought over from oslo-incubator? You'd have to know what the previous commit was that was synced. - Brant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On Wed, Mar 12, 2014 at 11:03 AM, Duncan Thomas wrote: > On 15 January 2014 18:53, Brant Knudson wrote: > > > At no point do I care what are the different commits that are being > brought > > in from oslo-incubator. If the commits are listed in the commit message > then > > I feel an obligation to verify that they got the right commits in the > > message and that takes extra time for no gain. > > I find that I very much *do* want a list of what changes have been > pulled in, so I've so idea of the intent of the changes. Some of the > OSLO changes can be large and complicated, and the more clues as to > why things changed, the better the chance I've got of spotting > breakages or differing assumptions between cinder and OSLO (of which > there have been a number) > > I don't very often verify that the version that has been pulled in is > the very latest or anything like that - generally I want to know: > One thing that I think we should be verifying is that the changes being brought over have actually been committed to oslo-incubator. I'm sure there have been times where someone eager to get the fix in has not waited for the oslo-incubator merge before syncing their change over. - What issue are you trying to fix by doing an update? (The fact OSLO > is ahead of us is rarely a good enough reason on its own to do an > update - preferably reference a specific bug that exists in cinder) > When I sync a change from oslo-incubator to fix a bug I put Closes-Bug on the commit message to indicate what bug is being fixed. If the sync tool was enhanced to pick out the *-Bug references from the oslo commits to include in the sync commit message that would be handy. > - What other incidental changes are being pulled in (by intent, not > just the code) > - If I'm unsure about one of the incidental changes, how do I go find > the history for it, with lots of searching (hense the commit ID or the > change ID) - this lets me find bugs, reviews etc > How does one get the list of commits that are being brought over from oslo-incubator? You'd have to know what the previous commit was that was synced. - Brant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On 15 January 2014 18:53, Brant Knudson wrote: > At no point do I care what are the different commits that are being brought > in from oslo-incubator. If the commits are listed in the commit message then > I feel an obligation to verify that they got the right commits in the > message and that takes extra time for no gain. I find that I very much *do* want a list of what changes have been pulled in, so I've so idea of the intent of the changes. Some of the OSLO changes can be large and complicated, and the more clues as to why things changed, the better the chance I've got of spotting breakages or differing assumptions between cinder and OSLO (of which there have been a number) I don't very often verify that the version that has been pulled in is the very latest or anything like that - generally I want to know: - What issue are you trying to fix by doing an update? (The fact OSLO is ahead of us is rarely a good enough reason on its own to do an update - preferably reference a specific bug that exists in cinder) - What other incidental changes are being pulled in (by intent, not just the code) - If I'm unsure about one of the incidental changes, how do I go find the history for it, with lots of searching (hense the commit ID or the change ID) - this lets me find bugs, reviews etc ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On Wed, Jan 15, 2014 at 1:53 PM, Brant Knudson wrote: > > Here's the process I use to review a oslo-incubator merge... since I'm > mostly reviewing keystone I'm going to use that as the project. > > 1) Make sure keystone master is at the latest, and that oslo-incubator is > at the right level (the commit hash if they mentioned it or latest) > 2) run update.py in oslo-incubator, make sure that there's changes in > keystone (maybe we have it already from a different review) > 3) revert changes in keystone and check out the review with git-review -d > 4) run update.py again and make sure there's no changes in keystone -- If > there's changes then I -1 the review, since I can't verify it was done > correctly. > 5) Then I go through the review and look for problems... changes that > aren't backwards compatible, new config options for the sample config file, > etc. > The first few steps of that seem like something that could be automated with a fairly simple script. > At no point do I care what are the different commits that are being > brought in from oslo-incubator. If the commits are listed in the commit > message then I feel an obligation to verify that they got the right commits > in the message and that takes extra time for no gain. > That makes sense to me. The request from the summit was to include the commits, but I think so far everyone is comfortable with backing that off to just say the most recent hash. Doug > > So I would prefer it if the message for syncs from oslo-incubator included > the commit hash and did not include a list of changes. > > If generating the list of changes was automated then I'd add a step to my > process to verify the commit message was the same. > > - Brant > > > > On Wed, Jan 15, 2014 at 6:44 AM, Doug Hellmann < > doug.hellm...@dreamhost.com> wrote: > >> >> >> >> On Wed, Jan 15, 2014 at 4:40 AM, Flavio Percoco wrote: >> >>> On 14/01/14 16:33 -0600, Ben Nemec wrote: >>> On 2014-01-14 15:26, Doug Hellmann wrote: In the release meeting today, Russell proposed that we at least include the hash of the HEAD when the merge is done, to indicate how far along the oslo changes are. More detail is obviously better. So, let's consider this as a new policy. Does anyone foresee issues with making this work? >>> >>> +1 from me. Lets keep it simple. I agree with you on the fact that we >>> should be focusing more on graduating modules from the incubator. >>> >>> >>> I don't see a problem with doing that, but I'm not clear on where we're including the hash. In the file itself, in a separate file, and/or in the commit message? Even if we do no more automation, having the commit hash of the last sync would be immensely helpful. Not having to comb through commit logs to figure out when the last sync happened would be fantastic. :-) >>> >>> We should keep it in the openstack-modules.conf file and put it in the >>> commit message as well. IMHO. >>> >> >> I was thinking the commit message, but if you see usefulness in including >> the conf file, we could do that, too. >> >> Doug >> >> >> >>> >>> FF >>> >>> >>> -Ben On Tue, Jan 14, 2014 at 4:23 AM, Flavio Percoco wrote: On 13/01/14 12:07 -0500, Doug Hellmann wrote: [.] I spent some time trying to think through how we could improve the update script for [1], and I'm stumped on how to figure out *accurately* what state the project repositories are in today. We can't just compute the hash of the modules in the project receiving copies, and then look for them in the oslo-incubator repo, because we modify the files as we copy them out (to update the import statements and replace "oslo" with the receiving project name in some places like config option defaults). We could undo those changes before computing the hash, but the problem is further complicated because syncs are not being done of all modules together. The common code in a project doesn't move forward in step with the oslo-incubator repository as a whole. For example, sometimes only the openstack /common/log.py module is copied and not all of openstack/common. So log.py might be newer than a lot of the rest of the oslo code. The problem is even worse for something like rpc, where it's possible that modules within the rpc package might not all be updated together.
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
Here's the process I use to review a oslo-incubator merge... since I'm mostly reviewing keystone I'm going to use that as the project. 1) Make sure keystone master is at the latest, and that oslo-incubator is at the right level (the commit hash if they mentioned it or latest) 2) run update.py in oslo-incubator, make sure that there's changes in keystone (maybe we have it already from a different review) 3) revert changes in keystone and check out the review with git-review -d 4) run update.py again and make sure there's no changes in keystone -- If there's changes then I -1 the review, since I can't verify it was done correctly. 5) Then I go through the review and look for problems... changes that aren't backwards compatible, new config options for the sample config file, etc. At no point do I care what are the different commits that are being brought in from oslo-incubator. If the commits are listed in the commit message then I feel an obligation to verify that they got the right commits in the message and that takes extra time for no gain. So I would prefer it if the message for syncs from oslo-incubator included the commit hash and did not include a list of changes. If generating the list of changes was automated then I'd add a step to my process to verify the commit message was the same. - Brant On Wed, Jan 15, 2014 at 6:44 AM, Doug Hellmann wrote: > > > > On Wed, Jan 15, 2014 at 4:40 AM, Flavio Percoco wrote: > >> On 14/01/14 16:33 -0600, Ben Nemec wrote: >> >>> On 2014-01-14 15:26, Doug Hellmann wrote: >>> >>>In the release meeting today, Russell proposed that we at least >>> include the >>>hash of the HEAD when the merge is done, to indicate how far along >>> the oslo >>>changes are. More detail is obviously better. >>>So, let's consider this as a new policy. Does anyone foresee >>> issues with >>>making this work? >>> >>> >> >> +1 from me. Lets keep it simple. I agree with you on the fact that we >> should be focusing more on graduating modules from the incubator. >> >> >> I don't see a problem with doing that, but I'm not clear on where we're >>> including the hash. In the file itself, in a separate file, and/or in >>> the >>> commit message? >>> >>> Even if we do no more automation, having the commit hash of the last >>> sync would >>> be immensely helpful. Not having to comb through commit logs to figure >>> out >>> when the last sync happened would be fantastic. :-) >>> >> >> We should keep it in the openstack-modules.conf file and put it in the >> commit message as well. IMHO. >> > > I was thinking the commit message, but if you see usefulness in including > the conf file, we could do that, too. > > Doug > > > >> >> FF >> >> >> >>> -Ben >>> >>> >>> >>>On Tue, Jan 14, 2014 at 4:23 AM, Flavio Percoco >>> wrote: >>> >>>On 13/01/14 12:07 -0500, Doug Hellmann wrote: >>>[.] >>> >>> >>> >>>I spent some time trying to think through how we could >>> improve the >>>update >>>script for [1], and I'm stumped on how to figure out >>> *accurately* >>>what state >>>the project repositories are in today. >>> >>>We can't just compute the hash of the modules in the project >>>receiving copies, >>>and then look for them in the oslo-incubator repo, because we >>>modify the files >>>as we copy them out (to update the import statements and >>> replace >>>"oslo" with >>>the receiving project name in some places like config option >>>defaults). >>> >>>We could undo those changes before computing the hash, but the >>>problem is >>>further complicated because syncs are not being done of all >>> modules >>>together. >>>The common code in a project doesn't move forward in step >>> with the >>>oslo-incubator repository as a whole. For example, sometimes >>> only >>>the openstack >>>/common/log.py module is copied and not all of >>> openstack/common. So >>>log.py >>>might be newer than a lot of the rest of the oslo code. The >>> problem >>>is even >>>worse for something like rpc, where it's possible that modules >>>within the rpc >>>package might not all be updated together. >>> >>>We could probably spend a lot of effort building a tool to >>> tell us >>>exactly what >>>the state of all of each common file is in each project, to >>> figure >>>out what >>>needs to be synced. I would much rather spend that effort on >>>turning the common >>>code into libraries, though. >>> >>>So, here's an alternative: >>> >>>1. Projects accept a full sync of Oslo soon, including adding >>> a >>>value in their >>>openstac
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On 2014-01-15 07:05, Flavio Percoco wrote: On 15/01/14 07:44 -0500, Doug Hellmann wrote: [...] I don't see a problem with doing that, but I'm not clear on where we're including the hash. In the file itself, in a separate file, and/or in the commit message? Even if we do no more automation, having the commit hash of the last sync would be immensely helpful. Not having to comb through commit logs to figure out when the last sync happened would be fantastic. :-) We should keep it in the openstack-modules.conf file and put it in the commit message as well. IMHO. I was thinking the commit message, but if you see usefulness in including the conf file, we could do that, too. As for now, the use I can think of is to know what was the last oslo-incubator commit that was synced. We could get the same info by using git log and copying the commit but it's easier to check that file. (Pretty much like submodules do). And if we ever need to do something with that, it'll be easier to get the sha from the config file. This argument violates YAGNI, though... :D To me, not having to search the target project's git log to find the last sync is enough need to justify it. Storing the last sync hash shouldn't be a significant amount of work if we're already going to be putting it in the commit hash, and if it makes syncs a little easier then I think it's absolutely worth it. -Ben ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On 15/01/14 07:44 -0500, Doug Hellmann wrote: [...] I don't see a problem with doing that, but I'm not clear on where we're including the hash. In the file itself, in a separate file, and/or in the commit message? Even if we do no more automation, having the commit hash of the last sync would be immensely helpful. Not having to comb through commit logs to figure out when the last sync happened would be fantastic. :-) We should keep it in the openstack-modules.conf file and put it in the commit message as well. IMHO. I was thinking the commit message, but if you see usefulness in including the conf file, we could do that, too. As for now, the use I can think of is to know what was the last oslo-incubator commit that was synced. We could get the same info by using git log and copying the commit but it's easier to check that file. (Pretty much like submodules do). And if we ever need to do something with that, it'll be easier to get the sha from the config file. This argument violates YAGNI, though... :D FF -- @flaper87 Flavio Percoco pgpg2_2UF3e1S.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On Wed, Jan 15, 2014 at 4:40 AM, Flavio Percoco wrote: > On 14/01/14 16:33 -0600, Ben Nemec wrote: > >> On 2014-01-14 15:26, Doug Hellmann wrote: >> >>In the release meeting today, Russell proposed that we at least >> include the >>hash of the HEAD when the merge is done, to indicate how far along the >> oslo >>changes are. More detail is obviously better. >>So, let's consider this as a new policy. Does anyone foresee >> issues with >>making this work? >> >> > > +1 from me. Lets keep it simple. I agree with you on the fact that we > should be focusing more on graduating modules from the incubator. > > > I don't see a problem with doing that, but I'm not clear on where we're >> including the hash. In the file itself, in a separate file, and/or in the >> commit message? >> >> Even if we do no more automation, having the commit hash of the last sync >> would >> be immensely helpful. Not having to comb through commit logs to figure >> out >> when the last sync happened would be fantastic. :-) >> > > We should keep it in the openstack-modules.conf file and put it in the > commit message as well. IMHO. > I was thinking the commit message, but if you see usefulness in including the conf file, we could do that, too. Doug > > FF > > > >> -Ben >> >> >> >>On Tue, Jan 14, 2014 at 4:23 AM, Flavio Percoco >> wrote: >> >>On 13/01/14 12:07 -0500, Doug Hellmann wrote: >>[.] >> >> >> >>I spent some time trying to think through how we could improve >> the >>update >>script for [1], and I'm stumped on how to figure out >> *accurately* >>what state >>the project repositories are in today. >> >>We can't just compute the hash of the modules in the project >>receiving copies, >>and then look for them in the oslo-incubator repo, because we >>modify the files >>as we copy them out (to update the import statements and >> replace >>"oslo" with >>the receiving project name in some places like config option >>defaults). >> >>We could undo those changes before computing the hash, but the >>problem is >>further complicated because syncs are not being done of all >> modules >>together. >>The common code in a project doesn't move forward in step with >> the >>oslo-incubator repository as a whole. For example, sometimes >> only >>the openstack >>/common/log.py module is copied and not all of >> openstack/common. So >>log.py >>might be newer than a lot of the rest of the oslo code. The >> problem >>is even >>worse for something like rpc, where it's possible that modules >>within the rpc >>package might not all be updated together. >> >>We could probably spend a lot of effort building a tool to >> tell us >>exactly what >>the state of all of each common file is in each project, to >> figure >>out what >>needs to be synced. I would much rather spend that effort on >>turning the common >>code into libraries, though. >> >>So, here's an alternative: >> >>1. Projects accept a full sync of Oslo soon, including adding a >>value in their >>openstack-common.conf indicating which commit in >> oslo-incubator is >>reflected in >>the sync. We'll try to make those commit messages as detailed >> as >>possible. >> >>2. We modify update.py to remove the option to update >> individual >>modules when >>copying from oslo-incubator. The new version would always >> apply all >>changes >>from the last merged commit, as a series of commits, to the >>receiving project. >>So if nova is out of step by 3 commits, then 3 new commits >> would be >>created in >>the branch by the person doing the update, each with the >> commit log >>message >>from the change in oslo-incubator. (This lock-step approach is >>necessary to >>have any hope of figuring out which commits are actually being >>synced, so the >>log messages are accurate.) >> >>In my experience, when syncing files from oslo, it'll most likely >>require syncing more than one module. There's been just *few* times >>where copying a module from oslo resulted in just that specific >> module >>being copied. >> >>All that to say that I agree with this point. >> >> >> >> >>3. The person proposing the merge into the project can decide >>whether to squash >>the commits, or leave them as separate reviews. >> >> >> >> >>
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On 14/01/14 16:33 -0600, Ben Nemec wrote: On 2014-01-14 15:26, Doug Hellmann wrote: In the release meeting today, Russell proposed that we at least include the hash of the HEAD when the merge is done, to indicate how far along the oslo changes are. More detail is obviously better. So, let's consider this as a new policy. Does anyone foresee issues with making this work? +1 from me. Lets keep it simple. I agree with you on the fact that we should be focusing more on graduating modules from the incubator. I don't see a problem with doing that, but I'm not clear on where we're including the hash. In the file itself, in a separate file, and/or in the commit message? Even if we do no more automation, having the commit hash of the last sync would be immensely helpful. Not having to comb through commit logs to figure out when the last sync happened would be fantastic. :-) We should keep it in the openstack-modules.conf file and put it in the commit message as well. IMHO. FF -Ben On Tue, Jan 14, 2014 at 4:23 AM, Flavio Percoco wrote: On 13/01/14 12:07 -0500, Doug Hellmann wrote: [.] I spent some time trying to think through how we could improve the update script for [1], and I'm stumped on how to figure out *accurately* what state the project repositories are in today. We can't just compute the hash of the modules in the project receiving copies, and then look for them in the oslo-incubator repo, because we modify the files as we copy them out (to update the import statements and replace "oslo" with the receiving project name in some places like config option defaults). We could undo those changes before computing the hash, but the problem is further complicated because syncs are not being done of all modules together. The common code in a project doesn't move forward in step with the oslo-incubator repository as a whole. For example, sometimes only the openstack /common/log.py module is copied and not all of openstack/common. So log.py might be newer than a lot of the rest of the oslo code. The problem is even worse for something like rpc, where it's possible that modules within the rpc package might not all be updated together. We could probably spend a lot of effort building a tool to tell us exactly what the state of all of each common file is in each project, to figure out what needs to be synced. I would much rather spend that effort on turning the common code into libraries, though. So, here's an alternative: 1. Projects accept a full sync of Oslo soon, including adding a value in their openstack-common.conf indicating which commit in oslo-incubator is reflected in the sync. We'll try to make those commit messages as detailed as possible. 2. We modify update.py to remove the option to update individual modules when copying from oslo-incubator. The new version would always apply all changes from the last merged commit, as a series of commits, to the receiving project. So if nova is out of step by 3 commits, then 3 new commits would be created in the branch by the person doing the update, each with the commit log message from the change in oslo-incubator. (This lock-step approach is necessary to have any hope of figuring out which commits are actually being synced, so the log messages are accurate.) In my experience, when syncing files from oslo, it'll most likely require syncing more than one module. There's been just *few* times where copying a module from oslo resulted in just that specific module being copied. All that to say that I agree with this point. 3. The person proposing the merge into the project can decide whether to squash the commits, or leave them as separate reviews. If we use relative imports for modules in oslo incubators (as mentioned in another email in this thread) and we *always* keep everything up to the latest. What about reconsidering using git submodules? AFAIR, the main issue with git submodules is that we wanted to support updating individual modules. If we remove that option, I think git submodules would work just fine. Am I missing something? Instead of hacking on update.py we could work on a migration plan out of it. A downside of
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On 2014-01-14 15:26, Doug Hellmann wrote: > In the release meeting today, Russell proposed that we at least include the > hash of the HEAD when the merge is done, to indicate how far along the oslo > changes are. More detail is obviously better. > > So, let's consider this as a new policy. Does anyone foresee issues with > making this work? > > Doug I don't see a problem with doing that, but I'm not clear on where we're including the hash. In the file itself, in a separate file, and/or in the commit message? Even if we do no more automation, having the commit hash of the last sync would be immensely helpful. Not having to comb through commit logs to figure out when the last sync happened would be fantastic. :-) -Ben > On Tue, Jan 14, 2014 at 4:23 AM, Flavio Percoco wrote: > On 13/01/14 12:07 -0500, Doug Hellmann wrote: > [.] > > I spent some time trying to think through how we could improve the update > script for [1], and I'm stumped on how to figure out *accurately* what state > the project repositories are in today. > > We can't just compute the hash of the modules in the project receiving copies, > and then look for them in the oslo-incubator repo, because we modify the files > as we copy them out (to update the import statements and replace "oslo" with > the receiving project name in some places like config option defaults). > > We could undo those changes before computing the hash, but the problem is > further complicated because syncs are not being done of all modules together. > The common code in a project doesn't move forward in step with the > oslo-incubator repository as a whole. For example, sometimes only the > openstack > /common/log.py module is copied and not all of openstack/common. So log.py > might be newer than a lot of the rest of the oslo code. The problem is even > worse for something like rpc, where it's possible that modules within the rpc > package might not all be updated together. > > We could probably spend a lot of effort building a tool to tell us exactly > what > the state of all of each common file is in each project, to figure out what > needs to be synced. I would much rather spend that effort on turning the > common > code into libraries, though. > > So, here's an alternative: > > 1. Projects accept a full sync of Oslo soon, including adding a value in their > openstack-common.conf indicating which commit in oslo-incubator is reflected > in > the sync. We'll try to make those commit messages as detailed as possible. > > 2. We modify update.py to remove the option to update individual modules when > copying from oslo-incubator. The new version would always apply all changes > from the last merged commit, as a series of commits, to the receiving project. > So if nova is out of step by 3 commits, then 3 new commits would be created in > the branch by the person doing the update, each with the commit log message > from the change in oslo-incubator. (This lock-step approach is necessary to > have any hope of figuring out which commits are actually being synced, so the > log messages are accurate.) In my experience, when syncing files from oslo, > it'll most likely > require syncing more than one module. There's been just *few* times > where copying a module from oslo resulted in just that specific module > being copied. > > All that to say that I agree with this point. > > 3. The person proposing the merge into the project can decide whether to > squash > the commits, or leave them as separate reviews. > > If we use relative imports for modules in oslo incubators (as > mentioned in another email in this thread) and we *always* keep > everything up to the latest. What about reconsidering using git > submodules? > > AFAIR, the main issue with git submodules is that we wanted to support > updating individual modules. If we remove that option, I think git > submodules would work just fine. Am I missing something? > > Instead of hacking on update.py we could work on a migration plan out > of it. > > A downside of using submodules is that when moving the reference in > the submodule, it won't be obvious why that's happening, which is > something we wanted to fix with update.py. It would be up to the > committer to write a good commit message or to get the messages out of > the submodule history. > > Another downside is that it would be hard to apply isolated patches on > stable branches for security issues or really awful bugs. > > I'm less convinced about submodules now but I'm leaving this in the > email in case someone wants to dig a bit deeper in the topic. > > I'm not entirely certain I like this approach myself, but it's the best I've > been able to come up with. It essentially gives us the current process, while > removing the ability to potentially take a version of a module without taking > its dependencies (allowing us to step forward, and track the commit messages > accurately). It will also produce results si
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
In the release meeting today, Russell proposed that we at least include the hash of the HEAD when the merge is done, to indicate how far along the oslo changes are. More detail is obviously better. So, let's consider this as a new policy. Does anyone foresee issues with making this work? Doug On Tue, Jan 14, 2014 at 4:23 AM, Flavio Percoco wrote: > On 13/01/14 12:07 -0500, Doug Hellmann wrote: > [.] > > >> I spent some time trying to think through how we could improve the update >> script for [1], and I'm stumped on how to figure out *accurately* what >> state >> the project repositories are in today. >> >> We can't just compute the hash of the modules in the project receiving >> copies, >> and then look for them in the oslo-incubator repo, because we modify the >> files >> as we copy them out (to update the import statements and replace "oslo" >> with >> the receiving project name in some places like config option defaults). >> >> We could undo those changes before computing the hash, but the problem is >> further complicated because syncs are not being done of all modules >> together. >> The common code in a project doesn't move forward in step with the >> oslo-incubator repository as a whole. For example, sometimes only the >> openstack >> /common/log.py module is copied and not all of openstack/common. So log.py >> might be newer than a lot of the rest of the oslo code. The problem is >> even >> worse for something like rpc, where it's possible that modules within the >> rpc >> package might not all be updated together. >> >> We could probably spend a lot of effort building a tool to tell us >> exactly what >> the state of all of each common file is in each project, to figure out >> what >> needs to be synced. I would much rather spend that effort on turning the >> common >> code into libraries, though. >> >> So, here's an alternative: >> >> 1. Projects accept a full sync of Oslo soon, including adding a value in >> their >> openstack-common.conf indicating which commit in oslo-incubator is >> reflected in >> the sync. We'll try to make those commit messages as detailed as possible. >> >> 2. We modify update.py to remove the option to update individual modules >> when >> copying from oslo-incubator. The new version would always apply all >> changes >> from the last merged commit, as a series of commits, to the receiving >> project. >> So if nova is out of step by 3 commits, then 3 new commits would be >> created in >> the branch by the person doing the update, each with the commit log >> message >> from the change in oslo-incubator. (This lock-step approach is necessary >> to >> have any hope of figuring out which commits are actually being synced, so >> the >> log messages are accurate.) >> > > In my experience, when syncing files from oslo, it'll most likely > require syncing more than one module. There's been just *few* times > where copying a module from oslo resulted in just that specific module > being copied. > > All that to say that I agree with this point. > > > >> 3. The person proposing the merge into the project can decide whether to >> squash >> the commits, or leave them as separate reviews. >> >> > > If we use relative imports for modules in oslo incubators (as > mentioned in another email in this thread) and we *always* keep > everything up to the latest. What about reconsidering using git > submodules? > > AFAIR, the main issue with git submodules is that we wanted to support > updating individual modules. If we remove that option, I think git > submodules would work just fine. Am I missing something? > > Instead of hacking on update.py we could work on a migration plan out > of it. > > A downside of using submodules is that when moving the reference in > the submodule, it won't be obvious why that's happening, which is > something we wanted to fix with update.py. It would be up to the > committer to write a good commit message or to get the messages out of > the submodule history. > > Another downside is that it would be hard to apply isolated patches on > stable branches for security issues or really awful bugs. > > I'm less convinced about submodules now but I'm leaving this in the > email in case someone wants to dig a bit deeper in the topic. > > > I'm not entirely certain I like this approach myself, but it's the best >> I've >> been able to come up with. It essentially gives us the current process, >> while >> removing the ability to potentially take a version of a module without >> taking >> its dependencies (allowing us to step forward, and track the commit >> messages >> accurately). It will also produce results similar to what we will have >> when all >> of this oslo code moves into separate libraries, where the changes to the >> library will be seen by the projects without any action at all on their >> part. >> > > After going through this for a bit, I agree with you. The goal > of the update script should be: > >- Sync modules from the current s
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On 13/01/14 12:07 -0500, Doug Hellmann wrote: [.] I spent some time trying to think through how we could improve the update script for [1], and I'm stumped on how to figure out *accurately* what state the project repositories are in today. We can't just compute the hash of the modules in the project receiving copies, and then look for them in the oslo-incubator repo, because we modify the files as we copy them out (to update the import statements and replace "oslo" with the receiving project name in some places like config option defaults). We could undo those changes before computing the hash, but the problem is further complicated because syncs are not being done of all modules together. The common code in a project doesn't move forward in step with the oslo-incubator repository as a whole. For example, sometimes only the openstack /common/log.py module is copied and not all of openstack/common. So log.py might be newer than a lot of the rest of the oslo code. The problem is even worse for something like rpc, where it's possible that modules within the rpc package might not all be updated together. We could probably spend a lot of effort building a tool to tell us exactly what the state of all of each common file is in each project, to figure out what needs to be synced. I would much rather spend that effort on turning the common code into libraries, though. So, here's an alternative: 1. Projects accept a full sync of Oslo soon, including adding a value in their openstack-common.conf indicating which commit in oslo-incubator is reflected in the sync. We'll try to make those commit messages as detailed as possible. 2. We modify update.py to remove the option to update individual modules when copying from oslo-incubator. The new version would always apply all changes from the last merged commit, as a series of commits, to the receiving project. So if nova is out of step by 3 commits, then 3 new commits would be created in the branch by the person doing the update, each with the commit log message from the change in oslo-incubator. (This lock-step approach is necessary to have any hope of figuring out which commits are actually being synced, so the log messages are accurate.) In my experience, when syncing files from oslo, it'll most likely require syncing more than one module. There's been just *few* times where copying a module from oslo resulted in just that specific module being copied. All that to say that I agree with this point. 3. The person proposing the merge into the project can decide whether to squash the commits, or leave them as separate reviews. If we use relative imports for modules in oslo incubators (as mentioned in another email in this thread) and we *always* keep everything up to the latest. What about reconsidering using git submodules? AFAIR, the main issue with git submodules is that we wanted to support updating individual modules. If we remove that option, I think git submodules would work just fine. Am I missing something? Instead of hacking on update.py we could work on a migration plan out of it. A downside of using submodules is that when moving the reference in the submodule, it won't be obvious why that's happening, which is something we wanted to fix with update.py. It would be up to the committer to write a good commit message or to get the messages out of the submodule history. Another downside is that it would be hard to apply isolated patches on stable branches for security issues or really awful bugs. I'm less convinced about submodules now but I'm leaving this in the email in case someone wants to dig a bit deeper in the topic. I'm not entirely certain I like this approach myself, but it's the best I've been able to come up with. It essentially gives us the current process, while removing the ability to potentially take a version of a module without taking its dependencies (allowing us to step forward, and track the commit messages accurately). It will also produce results similar to what we will have when all of this oslo code moves into separate libraries, where the changes to the library will be seen by the projects without any action at all on their part. After going through this for a bit, I agree with you. The goal of the update script should be: - Sync modules from the current state to the most updated version - Make sure the update information is not lost. If there's an oslo sync review without the commits shas + description, it simply means the committer amended the message. OTOH, it will also require spending time on update.py, instead of releasing a library from the incubator. And it doesn't really buy us that much in terms of making the sync happen more easily, other than a reliable way of having entirely accurate commit messages. Although it distracts us from our real goal - releasing libraries - I still think is necessary. We should probably just reduce the changes needed as much as possib
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On Mon, Jan 13, 2014 at 3:41 PM, Kevin L. Mitchell < kevin.mitch...@rackspace.com> wrote: > On Mon, 2014-01-13 at 12:07 -0500, Doug Hellmann wrote: > > We can't just compute the hash of the modules in the project receiving > > copies, and then look for them in the oslo-incubator repo, because we > > modify the files as we copy them out (to update the import statements > > and replace "oslo" with the receiving project name in some places like > > config option defaults). > > Why not embed the hash into the files as we copy them out, maybe as a > specially formatted comment at the end of the file? > The problem is much much easier for the second update. The real challenge is getting the first update right, when we don't know what the initial hash is. > > That said, I've always thought that oslo-incubator should be a library > we use directly, rather than a repository we copy out of. > We're working on that. That's another reason I don't really want to spend a lot of time making update.py fancier -- it takes away from time I could be spending on making the code into libraries in the first place. Doug > -- > Kevin L. Mitchell > Rackspace > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On Mon, 2014-01-13 at 12:07 -0500, Doug Hellmann wrote: > We can't just compute the hash of the modules in the project receiving > copies, and then look for them in the oslo-incubator repo, because we > modify the files as we copy them out (to update the import statements > and replace "oslo" with the receiving project name in some places like > config option defaults). Why not embed the hash into the files as we copy them out, maybe as a specially formatted comment at the end of the file? That said, I've always thought that oslo-incubator should be a library we use directly, rather than a repository we copy out of. -- Kevin L. Mitchell Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On Mon, Jan 13, 2014 at 1:16 PM, Robert Myers wrote: > We could always use relative imports in oslo :) Then you could put it > where ever you wanted to without needing to rewrite the import statements. > That may be a good idea, but doesn't really solve the problem at hand. Doug > > > On Mon, Jan 13, 2014 at 11:07 AM, Doug Hellmann < > doug.hellm...@dreamhost.com> wrote: > >> [resurrecting an old thread] >> >> >> On Wed, Nov 27, 2013 at 6:26 AM, Flavio Percoco wrote: >> >>> On 27/11/13 10:59 +, Mark McLoughlin wrote: >>> On Wed, 2013-11-27 at 11:50 +0100, Flavio Percoco wrote: > On 26/11/13 22:54 +, Mark McLoughlin wrote: > >On Fri, 2013-11-22 at 12:39 -0500, Doug Hellmann wrote: > >> On Fri, Nov 22, 2013 at 4:11 AM, Flavio Percoco > wrote: > >> >1) Store the commit sha from which the module was copied from. > >> >Every project using oslo, currently keeps the list of modules > it > >> >is using in `openstack-modules.conf` in a `module` parameter. > We > >> >could store, along with the module name, the sha of the commit > it > >> >was last synced from: > >> > > >> >module=log,commit > >> > > >> >or > >> > > >> >module=log > >> >log=commit > >> > > >> > >> The second form will be easier to manage. Humans edit the module > field and > >> the script will edit the others. > > > >How about adding it as a comment at the end of the python files > >themselves and leaving openstack-common.conf for human editing? > > I think having the commit sha will give us a starting point from which > we could start updating that module from. > Sure, my only point was about where the commit sha comes from - i.e. whether it's from a comment at the end of the python module itself or in openstack-common.conf >>> >>> And, indeed you said 'at the end of the python files'. Don't ask me >>> how the heck I misread that. >>> >>> The benefit I see from having them in the openstack-common.conf is >>> that we can register a `StrOpt` for each object dynamically and get >>> the sha using oslo.config. If we put it as a comment at the end of the >>> python file, we'll have to read it and 'parse' it, I guess. >>> >>> >>> It will mostly help with > getting a diff for that module and the short commit messages where it > was modified. > > Here's a pseudo-buggy-algorithm for the update process: > > (1) Get current sha for $module > (2) Get list of new commits for $module > (3) for each commit of $module: > (3.1) for each modified_module in $commit > (3.1.1) Update those modules up to $commit > (1)(modified_module) > (3.2) Copy the new file > (3.3) Update openstack-common with the latest sha > > This trusts the granularity and isolation of the patches proposed in > oslo-incubator. However, in cases like 'remove vim mode lines' it'll > fail assuming that updating every module is necessary - which is true > from a git stand point. > This is another variant of the kind of inter-module dependency smarts that update.py already has ... I'd be inclined to just omit those smarts and just require the caller to explicitly list the modules they want to include. Maybe update.py could include some reporting to help with that choice like "module foo depends on modules bar and blaa, maybe you want to include them too" and "commit XXX modified module foo, but also module bar and blaa, maybe you want to include them too". >>> >>> But, if we get to the point of suggesting the user to update module >>> foo because it was modified in commit XXX, we'd have everything needed >>> to make it recursive and update those modules as well. >>> >>> I agree with you on making it explicit, though. What about making it >>> interactive then? update.py could ask users if they want to update >>> module foo because it was modified in commit XXX and do it right away, >>> which is not very different from updating module foo, print a report >>> and let the user choose afterwards. >>> >>> (/me feels like Gollum now) >>> >>> I prefer the interactive way though, at least it doesn't require the >>> user to run update several times for each module. We could also add a >>> `--no-stop` flag that does exactly what you suggested. >>> >> >> I spent some time trying to think through how we could improve the update >> script for [1], and I'm stumped on how to figure out *accurately* what >> state the project repositories are in today. >> >> We can't just compute the hash of the modules in the project receiving >> copies, and then look for them in the oslo-incubator repo, because we >> modify the files as we copy them out (to update the import statements and >> replace "oslo" with the receiving proj
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
We could always use relative imports in oslo :) Then you could put it where ever you wanted to without needing to rewrite the import statements. On Mon, Jan 13, 2014 at 11:07 AM, Doug Hellmann wrote: > [resurrecting an old thread] > > > On Wed, Nov 27, 2013 at 6:26 AM, Flavio Percoco wrote: > >> On 27/11/13 10:59 +, Mark McLoughlin wrote: >> >>> On Wed, 2013-11-27 at 11:50 +0100, Flavio Percoco wrote: >>> On 26/11/13 22:54 +, Mark McLoughlin wrote: >On Fri, 2013-11-22 at 12:39 -0500, Doug Hellmann wrote: >> On Fri, Nov 22, 2013 at 4:11 AM, Flavio Percoco wrote: >> >1) Store the commit sha from which the module was copied from. >> >Every project using oslo, currently keeps the list of modules it >> >is using in `openstack-modules.conf` in a `module` parameter. We >> >could store, along with the module name, the sha of the commit it >> >was last synced from: >> > >> >module=log,commit >> > >> >or >> > >> >module=log >> >log=commit >> > >> >> The second form will be easier to manage. Humans edit the module field and >> the script will edit the others. > >How about adding it as a comment at the end of the python files >themselves and leaving openstack-common.conf for human editing? I think having the commit sha will give us a starting point from which we could start updating that module from. >>> >>> Sure, my only point was about where the commit sha comes from - i.e. >>> whether it's from a comment at the end of the python module itself or in >>> openstack-common.conf >>> >> >> And, indeed you said 'at the end of the python files'. Don't ask me >> how the heck I misread that. >> >> The benefit I see from having them in the openstack-common.conf is >> that we can register a `StrOpt` for each object dynamically and get >> the sha using oslo.config. If we put it as a comment at the end of the >> python file, we'll have to read it and 'parse' it, I guess. >> >> >> >>> It will mostly help with getting a diff for that module and the short commit messages where it was modified. Here's a pseudo-buggy-algorithm for the update process: (1) Get current sha for $module (2) Get list of new commits for $module (3) for each commit of $module: (3.1) for each modified_module in $commit (3.1.1) Update those modules up to $commit (1)(modified_module) (3.2) Copy the new file (3.3) Update openstack-common with the latest sha This trusts the granularity and isolation of the patches proposed in oslo-incubator. However, in cases like 'remove vim mode lines' it'll fail assuming that updating every module is necessary - which is true from a git stand point. >>> >>> This is another variant of the kind of inter-module dependency smarts >>> that update.py already has ... I'd be inclined to just omit those smarts >>> and just require the caller to explicitly list the modules they want to >>> include. >>> >>> Maybe update.py could include some reporting to help with that choice >>> like "module foo depends on modules bar and blaa, maybe you want to >>> include them too" and "commit XXX modified module foo, but also module >>> bar and blaa, maybe you want to include them too". >>> >> >> But, if we get to the point of suggesting the user to update module >> foo because it was modified in commit XXX, we'd have everything needed >> to make it recursive and update those modules as well. >> >> I agree with you on making it explicit, though. What about making it >> interactive then? update.py could ask users if they want to update >> module foo because it was modified in commit XXX and do it right away, >> which is not very different from updating module foo, print a report >> and let the user choose afterwards. >> >> (/me feels like Gollum now) >> >> I prefer the interactive way though, at least it doesn't require the >> user to run update several times for each module. We could also add a >> `--no-stop` flag that does exactly what you suggested. >> > > I spent some time trying to think through how we could improve the update > script for [1], and I'm stumped on how to figure out *accurately* what > state the project repositories are in today. > > We can't just compute the hash of the modules in the project receiving > copies, and then look for them in the oslo-incubator repo, because we > modify the files as we copy them out (to update the import statements and > replace "oslo" with the receiving project name in some places like config > option defaults). > > We could undo those changes before computing the hash, but the problem is > further complicated because syncs are not being done of all modules > together. The common code in a project doesn't move forward in step with > the oslo-incubator r
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
[resurrecting an old thread] On Wed, Nov 27, 2013 at 6:26 AM, Flavio Percoco wrote: > On 27/11/13 10:59 +, Mark McLoughlin wrote: > >> On Wed, 2013-11-27 at 11:50 +0100, Flavio Percoco wrote: >> >>> On 26/11/13 22:54 +, Mark McLoughlin wrote: >>> >On Fri, 2013-11-22 at 12:39 -0500, Doug Hellmann wrote: >>> >> On Fri, Nov 22, 2013 at 4:11 AM, Flavio Percoco >>> wrote: >>> >> >1) Store the commit sha from which the module was copied from. >>> >> >Every project using oslo, currently keeps the list of modules it >>> >> >is using in `openstack-modules.conf` in a `module` parameter. We >>> >> >could store, along with the module name, the sha of the commit it >>> >> >was last synced from: >>> >> > >>> >> >module=log,commit >>> >> > >>> >> >or >>> >> > >>> >> >module=log >>> >> >log=commit >>> >> > >>> >> >>> >> The second form will be easier to manage. Humans edit the module >>> field and >>> >> the script will edit the others. >>> > >>> >How about adding it as a comment at the end of the python files >>> >themselves and leaving openstack-common.conf for human editing? >>> >>> I think having the commit sha will give us a starting point from which >>> we could start updating that module from. >>> >> >> Sure, my only point was about where the commit sha comes from - i.e. >> whether it's from a comment at the end of the python module itself or in >> openstack-common.conf >> > > And, indeed you said 'at the end of the python files'. Don't ask me > how the heck I misread that. > > The benefit I see from having them in the openstack-common.conf is > that we can register a `StrOpt` for each object dynamically and get > the sha using oslo.config. If we put it as a comment at the end of the > python file, we'll have to read it and 'parse' it, I guess. > > > >> It will mostly help with >>> getting a diff for that module and the short commit messages where it >>> was modified. >>> >>> Here's a pseudo-buggy-algorithm for the update process: >>> >>> (1) Get current sha for $module >>> (2) Get list of new commits for $module >>> (3) for each commit of $module: >>> (3.1) for each modified_module in $commit >>> (3.1.1) Update those modules up to $commit >>> (1)(modified_module) >>> (3.2) Copy the new file >>> (3.3) Update openstack-common with the latest sha >>> >>> This trusts the granularity and isolation of the patches proposed in >>> oslo-incubator. However, in cases like 'remove vim mode lines' it'll >>> fail assuming that updating every module is necessary - which is true >>> from a git stand point. >>> >> >> This is another variant of the kind of inter-module dependency smarts >> that update.py already has ... I'd be inclined to just omit those smarts >> and just require the caller to explicitly list the modules they want to >> include. >> >> Maybe update.py could include some reporting to help with that choice >> like "module foo depends on modules bar and blaa, maybe you want to >> include them too" and "commit XXX modified module foo, but also module >> bar and blaa, maybe you want to include them too". >> > > But, if we get to the point of suggesting the user to update module > foo because it was modified in commit XXX, we'd have everything needed > to make it recursive and update those modules as well. > > I agree with you on making it explicit, though. What about making it > interactive then? update.py could ask users if they want to update > module foo because it was modified in commit XXX and do it right away, > which is not very different from updating module foo, print a report > and let the user choose afterwards. > > (/me feels like Gollum now) > > I prefer the interactive way though, at least it doesn't require the > user to run update several times for each module. We could also add a > `--no-stop` flag that does exactly what you suggested. > I spent some time trying to think through how we could improve the update script for [1], and I'm stumped on how to figure out *accurately* what state the project repositories are in today. We can't just compute the hash of the modules in the project receiving copies, and then look for them in the oslo-incubator repo, because we modify the files as we copy them out (to update the import statements and replace "oslo" with the receiving project name in some places like config option defaults). We could undo those changes before computing the hash, but the problem is further complicated because syncs are not being done of all modules together. The common code in a project doesn't move forward in step with the oslo-incubator repository as a whole. For example, sometimes only the openstack/common/log.py module is copied and not all of openstack/common. So log.py might be newer than a lot of the rest of the oslo code. The problem is even worse for something like rpc, where it's possible that modules within the rpc package might not all be updated together. We
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On 27/11/13 10:59 +, Mark McLoughlin wrote: On Wed, 2013-11-27 at 11:50 +0100, Flavio Percoco wrote: On 26/11/13 22:54 +, Mark McLoughlin wrote: >On Fri, 2013-11-22 at 12:39 -0500, Doug Hellmann wrote: >> On Fri, Nov 22, 2013 at 4:11 AM, Flavio Percoco wrote: >> >1) Store the commit sha from which the module was copied from. >> >Every project using oslo, currently keeps the list of modules it >> >is using in `openstack-modules.conf` in a `module` parameter. We >> >could store, along with the module name, the sha of the commit it >> >was last synced from: >> > >> >module=log,commit >> > >> >or >> > >> >module=log >> >log=commit >> > >> >> The second form will be easier to manage. Humans edit the module field and >> the script will edit the others. > >How about adding it as a comment at the end of the python files >themselves and leaving openstack-common.conf for human editing? I think having the commit sha will give us a starting point from which we could start updating that module from. Sure, my only point was about where the commit sha comes from - i.e. whether it's from a comment at the end of the python module itself or in openstack-common.conf And, indeed you said 'at the end of the python files'. Don't ask me how the heck I misread that. The benefit I see from having them in the openstack-common.conf is that we can register a `StrOpt` for each object dynamically and get the sha using oslo.config. If we put it as a comment at the end of the python file, we'll have to read it and 'parse' it, I guess. It will mostly help with getting a diff for that module and the short commit messages where it was modified. Here's a pseudo-buggy-algorithm for the update process: (1) Get current sha for $module (2) Get list of new commits for $module (3) for each commit of $module: (3.1) for each modified_module in $commit (3.1.1) Update those modules up to $commit (1)(modified_module) (3.2) Copy the new file (3.3) Update openstack-common with the latest sha This trusts the granularity and isolation of the patches proposed in oslo-incubator. However, in cases like 'remove vim mode lines' it'll fail assuming that updating every module is necessary - which is true from a git stand point. This is another variant of the kind of inter-module dependency smarts that update.py already has ... I'd be inclined to just omit those smarts and just require the caller to explicitly list the modules they want to include. Maybe update.py could include some reporting to help with that choice like "module foo depends on modules bar and blaa, maybe you want to include them too" and "commit XXX modified module foo, but also module bar and blaa, maybe you want to include them too". But, if we get to the point of suggesting the user to update module foo because it was modified in commit XXX, we'd have everything needed to make it recursive and update those modules as well. I agree with you on making it explicit, though. What about making it interactive then? update.py could ask users if they want to update module foo because it was modified in commit XXX and do it right away, which is not very different from updating module foo, print a report and let the user choose afterwards. (/me feels like Gollum now) I prefer the interactive way though, at least it doesn't require the user to run update several times for each module. We could also add a `--no-stop` flag that does exactly what you suggested. Cheers, FF -- @flaper87 Flavio Percoco pgpjWZxq5ji2s.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On Wed, 2013-11-27 at 11:50 +0100, Flavio Percoco wrote: > On 26/11/13 22:54 +, Mark McLoughlin wrote: > >On Fri, 2013-11-22 at 12:39 -0500, Doug Hellmann wrote: > >> On Fri, Nov 22, 2013 at 4:11 AM, Flavio Percoco wrote: > >> >1) Store the commit sha from which the module was copied from. > >> >Every project using oslo, currently keeps the list of modules it > >> >is using in `openstack-modules.conf` in a `module` parameter. We > >> >could store, along with the module name, the sha of the commit it > >> >was last synced from: > >> > > >> >module=log,commit > >> > > >> >or > >> > > >> >module=log > >> >log=commit > >> > > >> > >> The second form will be easier to manage. Humans edit the module field and > >> the script will edit the others. > > > >How about adding it as a comment at the end of the python files > >themselves and leaving openstack-common.conf for human editing? > > I think having the commit sha will give us a starting point from which > we could start updating that module from. Sure, my only point was about where the commit sha comes from - i.e. whether it's from a comment at the end of the python module itself or in openstack-common.conf > It will mostly help with > getting a diff for that module and the short commit messages where it > was modified. > > Here's a pseudo-buggy-algorithm for the update process: > > (1) Get current sha for $module > (2) Get list of new commits for $module > (3) for each commit of $module: > (3.1) for each modified_module in $commit > (3.1.1) Update those modules up to $commit (1)(modified_module) > (3.2) Copy the new file > (3.3) Update openstack-common with the latest sha > > This trusts the granularity and isolation of the patches proposed in > oslo-incubator. However, in cases like 'remove vim mode lines' it'll > fail assuming that updating every module is necessary - which is true > from a git stand point. This is another variant of the kind of inter-module dependency smarts that update.py already has ... I'd be inclined to just omit those smarts and just require the caller to explicitly list the modules they want to include. Maybe update.py could include some reporting to help with that choice like "module foo depends on modules bar and blaa, maybe you want to include them too" and "commit XXX modified module foo, but also module bar and blaa, maybe you want to include them too". Mark. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On 26/11/13 22:54 +, Mark McLoughlin wrote: On Fri, 2013-11-22 at 12:39 -0500, Doug Hellmann wrote: On Fri, Nov 22, 2013 at 4:11 AM, Flavio Percoco wrote: >1) Store the commit sha from which the module was copied from. >Every project using oslo, currently keeps the list of modules it >is using in `openstack-modules.conf` in a `module` parameter. We >could store, along with the module name, the sha of the commit it >was last synced from: > >module=log,commit > >or > >module=log >log=commit > The second form will be easier to manage. Humans edit the module field and the script will edit the others. How about adding it as a comment at the end of the python files themselves and leaving openstack-common.conf for human editing? I think having the commit sha will give us a starting point from which we could start updating that module from. It will mostly help with getting a diff for that module and the short commit messages where it was modified. Here's a pseudo-buggy-algorithm for the update process: (1) Get current sha for $module (2) Get list of new commits for $module (3) for each commit of $module: (3.1) for each modified_module in $commit (3.1.1) Update those modules up to $commit (1)(modified_module) (3.2) Copy the new file (3.3) Update openstack-common with the latest sha This trusts the granularity and isolation of the patches proposed in oslo-incubator. However, in cases like 'remove vim mode lines' it'll fail assuming that updating every module is necessary - which is true from a git stand point. The above will make update.py way smarter - and complex - than it's today, but given the number of projects - new and existing - we have now, I think it's worth it. Thoughts? FF -- @flaper87 Flavio Percoco pgptFgZslWEs0.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On Fri, 2013-11-22 at 16:24 +, Duncan Thomas wrote: > On 22 November 2013 14:59, Ben Nemec wrote: > > > One other thought I had was to add the ability to split one Oslo sync up > > into multiple commits, either one per module, or even one per Oslo commit > > for some really large module changes (I'm thinking of the 1000 line db sync > > someone mentioned recently). It would be more review churn, but at least it > > would keep the changes per review down to a more reasonable level. I'm not > > positive it would be beneficial, but I thought I'd mention it. > > Cinder (often but not always me) tends to reject merges that do more > that one module at a time, because it makes it far harder to review > and spot problems, so some from of automation of this would be great. Yes, it's good practice to sync related changes in individual commits, rather than one big oslo sync. An example from a previous sync I did in Cinder: https://review.openstack.org/#/c/12409/ (cfg) https://review.openstack.org/#/c/12411/ (zmq) https://review.openstack.org/#/c/12410/ (notifier) https://review.openstack.org/#/c/12412/ (misc) These were all proposed at the same time, but I split related notable changes into their own commits and then had a "misc" sync for more trivial stuff. Mark. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On Fri, 2013-11-22 at 12:39 -0500, Doug Hellmann wrote: > On Fri, Nov 22, 2013 at 4:11 AM, Flavio Percoco wrote: > > > Greetings, > > > > Based on the recent discussion that came out about not having enough > > information in the commit message when syncing oslo-incubator modules, > > I was thinking that besides encouraging people to write better commit > > messages, we could also improve the script we use to sync those > > modules. > > > > Some of the changes that I've been thinking of: > > > >1) Store the commit sha from which the module was copied from. > >Every project using oslo, currently keeps the list of modules it > >is using in `openstack-modules.conf` in a `module` parameter. We > >could store, along with the module name, the sha of the commit it > >was last synced from: > > > >module=log,commit > > > >or > > > >module=log > >log=commit > > > > The second form will be easier to manage. Humans edit the module field and > the script will edit the others. How about adding it as a comment at the end of the python files themselves and leaving openstack-common.conf for human editing? Mark. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
Hello. On Fri, Nov 22, 2013 at 1:11 PM, Flavio Percoco wrote: >1) Store the commit sha from which the module was copied from. > I would suggest we don't duplicate this in every project's repo. It is possible to find using the contents of the modules currently in project's repo the corresponding commit from oslo-incubator (see [1] for a hint, feel free to ask me about Git details). So there's no need to store this information in a separate file. 2) Add an 'auto-commit' parameter to the update script that will >generate a commit message with the short log of the commits where >the modules being updated were modified. Soemthing like: > >Syncing oslo-incubator modules > >log.py: >commit1: short-message >commit2: short-message >commit3: short-message > >lockutils: >commit4: short-message >commit5: short-message > The script can use this information as well to get the last synced commit for each module. We should to agree to use this format for all commit messages and make commit detection easier. I would drop the colon though to make it look like the usual 'git log --oneline' output. [1] http://stackoverflow.com/questions/223678/which-commit-has-this-blob -- Kind regards, Yuriy. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On 23/11/13 11:54 -0500, Doug Hellmann wrote: Thanks for the reminder, Sandy. https://bugs.launchpad.net/oslo/+bug/1254300 On Sat, Nov 23, 2013 at 9:39 AM, Sandy Walsh wrote: Seeing this thread reminded me: We need support in the update script for entry points in olso setup.cfg to make their way into the target project. So, if update is getting some love, please keep that in mind. Good point, +1 I created this blueprint[0] and assigned it to me. I'll give some love to update.py and get it done for I-2. Cheers, FF [0] https://blueprints.launchpad.net/oslo/+spec/improve-update-script -- @flaper87 Flavio Percoco pgp610yEgnGwn.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On 25/11/13 09:34 +0800, Zhongyue Luo wrote: Just a thought but couldn't the changes of a module update be calculated by comparing the last commit dates of the source and target module? For instance, if module A's update patch for Nova was uploaded on date XX then we can filter out the changes from XX~present and print it out for the author to paste in the commit message when running update.py for module A. This way we might not need any changes to the openstack-modules.conf? Not sure that would work. The update date of module A in project X doesn't mean the project is at its latest version. I could've updated an older version because the latest would've broken project X. Cheers, FF On Sun, Nov 24, 2013 at 12:54 AM, Doug Hellmann wrote: Thanks for the reminder, Sandy. https://bugs.launchpad.net/oslo/+bug/1254300 On Sat, Nov 23, 2013 at 9:39 AM, Sandy Walsh wrote: Seeing this thread reminded me: We need support in the update script for entry points in olso setup.cfg to make their way into the target project. So, if update is getting some love, please keep that in mind. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Intel SSG/STO/DCST/CIT 880 Zixing Road, Zizhu Science Park, Minhang District, 200241, Shanghai, China +862161166500 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco pgpxpMTb75cQH.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
Just a thought but couldn't the changes of a module update be calculated by comparing the last commit dates of the source and target module? For instance, if module A's update patch for Nova was uploaded on date XX then we can filter out the changes from XX~present and print it out for the author to paste in the commit message when running update.py for module A. This way we might not need any changes to the openstack-modules.conf? On Sun, Nov 24, 2013 at 12:54 AM, Doug Hellmann wrote: > Thanks for the reminder, Sandy. > > https://bugs.launchpad.net/oslo/+bug/1254300 > > > On Sat, Nov 23, 2013 at 9:39 AM, Sandy Walsh wrote: > >> Seeing this thread reminded me: >> >> We need support in the update script for entry points in olso setup.cfg >> to make their way into the target project. >> >> So, if update is getting some love, please keep that in mind. >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- *Intel SSG/STO/DCST/CIT* 880 Zixing Road, Zizhu Science Park, Minhang District, 200241, Shanghai, China +862161166500 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
Thanks for the reminder, Sandy. https://bugs.launchpad.net/oslo/+bug/1254300 On Sat, Nov 23, 2013 at 9:39 AM, Sandy Walsh wrote: > Seeing this thread reminded me: > > We need support in the update script for entry points in olso setup.cfg to > make their way into the target project. > > So, if update is getting some love, please keep that in mind. > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
Seeing this thread reminded me: We need support in the update script for entry points in olso setup.cfg to make their way into the target project. So, if update is getting some love, please keep that in mind. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On Fri, Nov 22, 2013 at 11:24 AM, Duncan Thomas wrote: > On 22 November 2013 14:59, Ben Nemec wrote: > > > One other thought I had was to add the ability to split one Oslo sync up > > into multiple commits, either one per module, or even one per Oslo commit > > for some really large module changes (I'm thinking of the 1000 line db > sync > > someone mentioned recently). It would be more review churn, but at > least it > > would keep the changes per review down to a more reasonable level. I'm > not > > positive it would be beneficial, but I thought I'd mention it. > > Cinder (often but not always me) tends to reject merges that do more > that one module at a time, because it makes it far harder to review > and spot problems, so some from of automation of this would be great. > There are times when the commits are related, but in general this seems like a good practice. Doug > > -- > Duncan Thomas > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On Fri, Nov 22, 2013 at 4:11 AM, Flavio Percoco wrote: > Greetings, > > Based on the recent discussion that came out about not having enough > information in the commit message when syncing oslo-incubator modules, > I was thinking that besides encouraging people to write better commit > messages, we could also improve the script we use to sync those > modules. > > Some of the changes that I've been thinking of: > >1) Store the commit sha from which the module was copied from. >Every project using oslo, currently keeps the list of modules it >is using in `openstack-modules.conf` in a `module` parameter. We >could store, along with the module name, the sha of the commit it >was last synced from: > >module=log,commit > >or > >module=log >log=commit > The second form will be easier to manage. Humans edit the module field and the script will edit the others. > >2) Add an 'auto-commit' parameter to the update script that will >generate a commit message with the short log of the commits where >the modules being updated were modified. Soemthing like: > >Syncing oslo-incubator modules > >log.py: >commit1: short-message >commit2: short-message >commit3: short-message > >lockutils: >commit4: short-message >commit5: short-message > > #1 will help with figuring out when was the last time a module was > updated and what changes have been introduced since then. > > #2 won't be the recommended way for writing commit messages. Oslo > incubator syncs can have different side-effects in every project. > However, it could be useful for trivial syncs. > It looks like #2 would be a good way to start with commit messages, and just leave it up to the person doing the merge to add the "why" information. Doug > > Thoughts? > > Cheers, > FF > > -- > @flaper87 > Flavio Percoco > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On 22 November 2013 14:59, Ben Nemec wrote: > One other thought I had was to add the ability to split one Oslo sync up > into multiple commits, either one per module, or even one per Oslo commit > for some really large module changes (I'm thinking of the 1000 line db sync > someone mentioned recently). It would be more review churn, but at least it > would keep the changes per review down to a more reasonable level. I'm not > positive it would be beneficial, but I thought I'd mention it. Cinder (often but not always me) tends to reject merges that do more that one module at a time, because it makes it far harder to review and spot problems, so some from of automation of this would be great. -- Duncan Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
On 2013-11-22 03:11, Flavio Percoco wrote: Greetings, Based on the recent discussion that came out about not having enough information in the commit message when syncing oslo-incubator modules, I was thinking that besides encouraging people to write better commit messages, we could also improve the script we use to sync those modules. Some of the changes that I've been thinking of: 1) Store the commit sha from which the module was copied from. Every project using oslo, currently keeps the list of modules it is using in `openstack-modules.conf` in a `module` parameter. We could store, along with the module name, the sha of the commit it was last synced from: module=log,commit or module=log log=commit 2) Add an 'auto-commit' parameter to the update script that will generate a commit message with the short log of the commits where the modules being updated were modified. Soemthing like: Syncing oslo-incubator modules log.py: commit1: short-message commit2: short-message commit3: short-message lockutils: commit4: short-message commit5: short-message #1 will help with figuring out when was the last time a module was updated and what changes have been introduced since then. #2 won't be the recommended way for writing commit messages. Oslo incubator syncs can have different side-effects in every project. However, it could be useful for trivial syncs. Thoughts? +1. I've been thinking along the same lines. I always try to include the git oneline output in my Oslo syncs anyway, but it's not trivial to figure out the appropriate ones to include so doing it automatically would be fantastic. One other thought I had was to add the ability to split one Oslo sync up into multiple commits, either one per module, or even one per Oslo commit for some really large module changes (I'm thinking of the 1000 line db sync someone mentioned recently). It would be more review churn, but at least it would keep the changes per review down to a more reasonable level. I'm not positive it would be beneficial, but I thought I'd mention it. -Ben ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Oslo] Improving oslo-incubator update.py
Greetings, Based on the recent discussion that came out about not having enough information in the commit message when syncing oslo-incubator modules, I was thinking that besides encouraging people to write better commit messages, we could also improve the script we use to sync those modules. Some of the changes that I've been thinking of: 1) Store the commit sha from which the module was copied from. Every project using oslo, currently keeps the list of modules it is using in `openstack-modules.conf` in a `module` parameter. We could store, along with the module name, the sha of the commit it was last synced from: module=log,commit or module=log log=commit 2) Add an 'auto-commit' parameter to the update script that will generate a commit message with the short log of the commits where the modules being updated were modified. Soemthing like: Syncing oslo-incubator modules log.py: commit1: short-message commit2: short-message commit3: short-message lockutils: commit4: short-message commit5: short-message #1 will help with figuring out when was the last time a module was updated and what changes have been introduced since then. #2 won't be the recommended way for writing commit messages. Oslo incubator syncs can have different side-effects in every project. However, it could be useful for trivial syncs. Thoughts? Cheers, FF -- @flaper87 Flavio Percoco pgpCmV9OwmxXq.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
