Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2014-03-17 Thread Ben Nemec

On 2014-03-15 12:23, Solly Ross wrote:

--nodeps will only sync the modules specified on the command line:
https://wiki.openstack.org/wiki/Oslo#Syncing_Code_from_Incubator


Heh, whoops.  Must have missed that.  Is it in the README/info at the
top of the update.py script?


It wasn't, but I pushed a change to add it: 
https://review.openstack.org/#/c/81004/




That said, it's not always safe to do that.  You might sync a change 
in

one module that depends on a change in another module and end up
breaking something.  It might not be caught in the sync either because
the Oslo unit tests don't get synced across.


Hmm... I suppose this is why we have libraries with dependencies (not
meant to sound snarky).
Although in the case of updating a library that you wrote, it's less
likely to break things.


Yeah, ideally you would never need nodeps because the deps would already 
be up to date when you do your sync, but unfortunately we aren't that 
good about keeping Oslo syncs current right now.  Hopefully that will 
get better in Juno, but we'll see.




Best Regards,
Solly Ross

- Original Message -
From: Ben Nemec openst...@nemebean.com
To: OpenStack Development Mailing List (not for usage questions)
openstack-dev@lists.openstack.org
Cc: Solly Ross sr...@redhat.com
Sent: Friday, March 14, 2014 4:36:03 PM
Subject: Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

On 2014-03-14 14:49, Solly Ross wrote:

It would also be great if there was a way to only sync one package.


There is. :-)

--nodeps will only sync the modules specified on the command line:
https://wiki.openstack.org/wiki/Oslo#Syncing_Code_from_Incubator

That said, it's not always safe to do that.  You might sync a change in
one module that depends on a change in another module and end up
breaking something.  It might not be caught in the sync either because
the Oslo unit tests don't get synced across.


When adding a new library
to a project (e.g. openstack.common.report to Nova), one would want to
only sync the openstack.common.report
parts, and not the any changes from the rest of openstack.common.  My
process has been

1. Edit openstack-common.conf to only contain the packages I want
2. Run the update
3. Make sure there wasn't code that didn't get changed from
'openstack.common.xyz' to 'nova.openstack.common.xyz' (hint: this
happens some times)
4. git checkout openstack-common.conf to revert the changes to
openstack-common.conf

IMHO, update.py needs a bit of work (well, I think the whole code
copying thing needs a bit of work, but that's a different story).

Best Regards,
Solly Ross

- Original Message -
From: Jay S Bryant jsbry...@us.ibm.com
To: OpenStack Development Mailing List (not for usage questions)
openstack-dev@lists.openstack.org
Sent: Friday, March 14, 2014 3:36:49 PM
Subject: Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py




From: Brant Knudson b...@acm.org
To: OpenStack Development Mailing List (not for usage questions)
openstack-dev@lists.openstack.org,
Date: 03/14/2014 02:21 PM
Subject: Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py







On Fri, Mar 14, 2014 at 2:05 PM, Jay S Bryant  jsbry...@us.ibm.com 
wrote:

It would be great if we could get the process for this automated. In
the
mean time, those of us doing the syncs will just have to slog through
the
process.

Jay



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


What's the process? How do I generate the list of changes?

Brant
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Brant,

My process thus far has been the following:


1. Do the sync to see what files are changed.
2. Take a look at the last commit sync'd to what is currently in
master for a file.
3. Document all the commits that have come in on that file since.
4. Repeat process for all the relevant files if there is more than
one.
5. If are multiples files I organize the commits with a list of
the files touched by that commit.
6. Document the master level of Oslo when the sync was done for
reference.

Process may not be perfect, but it gets the job done. Here is an
example of the format I use: https://review.openstack.org/#/c/75740/

Jay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2014-03-15 Thread Solly Ross
 --nodeps will only sync the modules specified on the command line: 
 https://wiki.openstack.org/wiki/Oslo#Syncing_Code_from_Incubator

Heh, whoops.  Must have missed that.  Is it in the README/info at the top of 
the update.py script?

 That said, it's not always safe to do that.  You might sync a change in 
 one module that depends on a change in another module and end up 
 breaking something.  It might not be caught in the sync either because 
 the Oslo unit tests don't get synced across.

Hmm... I suppose this is why we have libraries with dependencies (not meant to 
sound snarky). 
Although in the case of updating a library that you wrote, it's less likely to 
break things.

Best Regards,
Solly Ross

- Original Message -
From: Ben Nemec openst...@nemebean.com
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org
Cc: Solly Ross sr...@redhat.com
Sent: Friday, March 14, 2014 4:36:03 PM
Subject: Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

On 2014-03-14 14:49, Solly Ross wrote:
 It would also be great if there was a way to only sync one package.

There is. :-)

--nodeps will only sync the modules specified on the command line: 
https://wiki.openstack.org/wiki/Oslo#Syncing_Code_from_Incubator

That said, it's not always safe to do that.  You might sync a change in 
one module that depends on a change in another module and end up 
breaking something.  It might not be caught in the sync either because 
the Oslo unit tests don't get synced across.

 When adding a new library
 to a project (e.g. openstack.common.report to Nova), one would want to
 only sync the openstack.common.report
 parts, and not the any changes from the rest of openstack.common.  My
 process has been
 
 1. Edit openstack-common.conf to only contain the packages I want
 2. Run the update
 3. Make sure there wasn't code that didn't get changed from
 'openstack.common.xyz' to 'nova.openstack.common.xyz' (hint: this
 happens some times)
 4. git checkout openstack-common.conf to revert the changes to
 openstack-common.conf
 
 IMHO, update.py needs a bit of work (well, I think the whole code
 copying thing needs a bit of work, but that's a different story).
 
 Best Regards,
 Solly Ross
 
 - Original Message -
 From: Jay S Bryant jsbry...@us.ibm.com
 To: OpenStack Development Mailing List (not for usage questions)
 openstack-dev@lists.openstack.org
 Sent: Friday, March 14, 2014 3:36:49 PM
 Subject: Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
 
 
 
 
 From: Brant Knudson b...@acm.org
 To: OpenStack Development Mailing List (not for usage questions)
 openstack-dev@lists.openstack.org,
 Date: 03/14/2014 02:21 PM
 Subject: Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py
 
 
 
 
 
 
 
 On Fri, Mar 14, 2014 at 2:05 PM, Jay S Bryant  jsbry...@us.ibm.com  
 wrote:
 
 It would be great if we could get the process for this automated. In 
 the
 mean time, those of us doing the syncs will just have to slog through 
 the
 process.
 
 Jay
 
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 What's the process? How do I generate the list of changes?
 
 Brant
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 Brant,
 
 My process thus far has been the following:
 
 
 1. Do the sync to see what files are changed.
 2. Take a look at the last commit sync'd to what is currently in
 master for a file.
 3. Document all the commits that have come in on that file since.
 4. Repeat process for all the relevant files if there is more than 
 one.
 5. If are multiples files I organize the commits with a list of
 the files touched by that commit.
 6. Document the master level of Oslo when the sync was done for 
 reference.
 
 Process may not be perfect, but it gets the job done. Here is an
 example of the format I use: https://review.openstack.org/#/c/75740/
 
 Jay
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2014-03-14 Thread Brant Knudson
On Wed, Mar 12, 2014 at 11:03 AM, Duncan Thomas duncan.tho...@gmail.comwrote:

 On 15 January 2014 18:53, Brant Knudson b...@acm.org wrote:

  At no point do I care what are the different commits that are being
 brought
  in from oslo-incubator. If the commits are listed in the commit message
 then
  I feel an obligation to verify that they got the right commits in the
  message and that takes extra time for no gain.

 I find that I very much *do* want a list of what changes have been
 pulled in, so I've so idea of the intent of the changes. Some of the
 OSLO changes can be large and complicated, and the more clues as to
 why things changed, the better the chance I've got of spotting
 breakages or differing assumptions between cinder and OSLO (of which
 there have been a number)

 I don't very often verify that the version that has been pulled in is
 the very latest or anything like that - generally I want to know:


One thing that I think we should be verifying is that the changes being
brought over have actually been committed to oslo-incubator. I'm sure there
have been times where someone eager to get the fix in has not waited for
the oslo-incubator merge before syncing their change over.

 - What issue are you trying to fix by doing an update? (The fact OSLO
 is ahead of us is rarely a good enough reason on its own to do an
 update - preferably reference a specific bug that exists in cinder)


When I sync a change from oslo-incubator to fix a bug I put Closes-Bug on
the commit message to indicate what bug is being fixed. If the sync tool
was enhanced to pick out the *-Bug references from the oslo commits to
include in the sync commit message that would be handy.


  - What other incidental changes are being pulled in (by intent, not
 just the code)
  - If I'm unsure about one of the incidental changes, how do I go find
 the history for it, with lots of searching (hense the commit ID or the
 change ID) - this lets me find bugs, reviews etc


How does one get the list of commits that are being brought over from
oslo-incubator? You'd have to know what the previous commit was that was
synced.

- Brant
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2014-03-14 Thread Jay S Bryant
From:   Brant Knudson b...@acm.org
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org, 
Date:   03/14/2014 10:26 AM
Subject:Re: [openstack-dev] [Oslo] Improving oslo-incubator 
update.py





On Wed, Mar 12, 2014 at 11:03 AM, Duncan Thomas duncan.tho...@gmail.com 
wrote:
On 15 January 2014 18:53, Brant Knudson b...@acm.org wrote:

 At no point do I care what are the different commits that are being 
brought
 in from oslo-incubator. If the commits are listed in the commit message 
then
 I feel an obligation to verify that they got the right commits in the
 message and that takes extra time for no gain.

-- Duncan, It is important to know what commits are being brought over to 
help provide a pointer to
-- the possible cause of subsequent bugs that arise.  I.E. if we sync up 
the DB, there is a commit for fixing
-- db connection order and suddenly we are getting intermittent DB 
connection failures, it give us
-- a starting point to fixing the issue.

I find that I very much *do* want a list of what changes have been
pulled in, so I've so idea of the intent of the changes. Some of the
OSLO changes can be large and complicated, and the more clues as to
why things changed, the better the chance I've got of spotting
breakages or differing assumptions between cinder and OSLO (of which
there have been a number)

I don't very often verify that the version that has been pulled in is
the very latest or anything like that - generally I want to know:

One thing that I think we should be verifying is that the changes being 
brought over have actually been committed to oslo-incubator. I'm sure 
there have been times where someone eager to get the fix in has not waited 
for the oslo-incubator merge before syncing their change over.

 - What issue are you trying to fix by doing an update? (The fact OSLO
is ahead of us is rarely a good enough reason on its own to do an
update - preferably reference a specific bug that exists in cinder)

When I sync a change from oslo-incubator to fix a bug I put Closes-Bug on 
the commit message to indicate what bug is being fixed. If the sync tool 
was enhanced to pick out the *-Bug references from the oslo commits to 
include in the sync commit message that would be handy.
 
 - What other incidental changes are being pulled in (by intent, not
just the code)
 - If I'm unsure about one of the incidental changes, how do I go find
the history for it, with lots of searching (hense the commit ID or the
change ID) - this lets me find bugs, reviews etc

How does one get the list of commits that are being brought over from 
oslo-incubator? You'd have to know what the previous commit was that was 
synced.

- Brant
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2014-03-14 Thread Duncan Thomas
On 14 March 2014 16:10, Jay S Bryant jsbry...@us.ibm.com wrote:
 -- Duncan, It is important to know what commits are being brought over to
 help provide a pointer to
 -- the possible cause of subsequent bugs that arise.  I.E. if we sync up
 the DB, there is a commit for fixing
 -- db connection order and suddenly we are getting intermittent DB
 connection failures, it give us
 -- a starting point to fixing the issue.


Jay, there's been a mix-up in who's saying what here. I *very much*
want to know what commits are being bought over. For slightly
different reasons (I'm mostly wanting them for easy review, you for
bug fixing). Brant is suggesting that just the last commit ID is
enough, which I disagree with (and will continue to hit -1/-2 for).

If somebody was to improve the import script to do this automatically
that would be great. Currently I can't see an easy way of
programatically telling when the last import was - I'll take another
look at the problem if somebody smarter than me doesn't sort it first

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2014-03-14 Thread Jay S Bryant
From:   Duncan Thomas duncan.tho...@gmail.com
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org, 
Date:   03/14/2014 11:49 AM
Subject:Re: [openstack-dev] [Oslo] Improving oslo-incubator 
update.py



On 14 March 2014 16:10, Jay S Bryant jsbry...@us.ibm.com wrote:
 -- Duncan, It is important to know what commits are being brought over 
to
 help provide a pointer to
 -- the possible cause of subsequent bugs that arise.  I.E. if we sync 
up
 the DB, there is a commit for fixing
 -- db connection order and suddenly we are getting intermittent DB
 connection failures, it give us
 -- a starting point to fixing the issue.


Jay, there's been a mix-up in who's saying what here. I *very much*
want to know what commits are being bought over. For slightly
different reasons (I'm mostly wanting them for easy review, you for
bug fixing). Brant is suggesting that just the last commit ID is
enough, which I disagree with (and will continue to hit -1/-2 for).

If somebody was to improve the import script to do this automatically
that would be great. Currently I can't see an easy way of
programatically telling when the last import was - I'll take another
look at the problem if somebody smarter than me doesn't sort it first

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Duncan,

Thanks for clarifying!  I was quite confused as I thought you were in
favor of the details in the commit messages.  Sorry for confusing who was 
carrying what stance forward.

Brant,

I agree with Duncan.  I think that having details of the commits that
are being pulled in with a sync commit is important.  I realize that the
user probably isn't going to do a perfect job, but it certainly helps 
to give context to the changes being made and provides important bread 
crumbs
for debug.

It would be great if we could get the process for this automated.  In the 
mean time, those of us doing the syncs will just have to slog through the
process.

Jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2014-03-14 Thread Jay S Bryant
From:   Brant Knudson b...@acm.org
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org, 
Date:   03/14/2014 02:21 PM
Subject:Re: [openstack-dev] [Oslo] Improving oslo-incubator 
update.py






On Fri, Mar 14, 2014 at 2:05 PM, Jay S Bryant jsbry...@us.ibm.com wrote:

It would be great if we could get the process for this automated.  In the 
mean time, those of us doing the syncs will just have to slog through the 
process. 

Jay



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


What's the process? How do I generate the list of changes?

Brant
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Brant,

My process thus far has been the following:

Do the sync to see what files are changed.
Take a look at the last commit sync'd to what is currently in master for a 
file.
Document all the commits that have come in on that file since.
Repeat process for all the relevant files if there is more than one.
If are multiples files I organize the commits with a list of the files 
touched by that commit.
Document the master level of Oslo when the sync was done for reference.

Process may not be perfect, but it gets the job done.  Here is an example 
of the format I use:  https://review.openstack.org/#/c/75740/

Jay
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2014-03-14 Thread Solly Ross
It would also be great if there was a way to only sync one package.  When 
adding a new library
to a project (e.g. openstack.common.report to Nova), one would want to only 
sync the openstack.common.report
parts, and not the any changes from the rest of openstack.common.  My process 
has been

1. Edit openstack-common.conf to only contain the packages I want
2. Run the update
3. Make sure there wasn't code that didn't get changed from 
'openstack.common.xyz' to 'nova.openstack.common.xyz' (hint: this happens some 
times)
4. git checkout openstack-common.conf to revert the changes to 
openstack-common.conf

IMHO, update.py needs a bit of work (well, I think the whole code copying thing 
needs a bit of work, but that's a different story).

Best Regards,
Solly Ross

- Original Message -
From: Jay S Bryant jsbry...@us.ibm.com
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org
Sent: Friday, March 14, 2014 3:36:49 PM
Subject: Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py




From: Brant Knudson b...@acm.org 
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org, 
Date: 03/14/2014 02:21 PM 
Subject: Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py 







On Fri, Mar 14, 2014 at 2:05 PM, Jay S Bryant  jsbry...@us.ibm.com  wrote: 

It would be great if we could get the process for this automated. In the 
mean time, those of us doing the syncs will just have to slog through the 
process. 

Jay 



___ 
OpenStack-dev mailing list 
OpenStack-dev@lists.openstack.org 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 


What's the process? How do I generate the list of changes? 

Brant 
___ 
OpenStack-dev mailing list 
OpenStack-dev@lists.openstack.org 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 


Brant, 

My process thus far has been the following: 


1. Do the sync to see what files are changed. 
2. Take a look at the last commit sync'd to what is currently in master for 
a file. 
3. Document all the commits that have come in on that file since. 
4. Repeat process for all the relevant files if there is more than one. 
5. If are multiples files I organize the commits with a list of the files 
touched by that commit. 
6. Document the master level of Oslo when the sync was done for reference. 

Process may not be perfect, but it gets the job done. Here is an example of the 
format I use: https://review.openstack.org/#/c/75740/ 

Jay 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2014-03-14 Thread Ben Nemec

On 2014-03-14 14:49, Solly Ross wrote:

It would also be great if there was a way to only sync one package.


There is. :-)

--nodeps will only sync the modules specified on the command line: 
https://wiki.openstack.org/wiki/Oslo#Syncing_Code_from_Incubator


That said, it's not always safe to do that.  You might sync a change in 
one module that depends on a change in another module and end up 
breaking something.  It might not be caught in the sync either because 
the Oslo unit tests don't get synced across.



When adding a new library
to a project (e.g. openstack.common.report to Nova), one would want to
only sync the openstack.common.report
parts, and not the any changes from the rest of openstack.common.  My
process has been

1. Edit openstack-common.conf to only contain the packages I want
2. Run the update
3. Make sure there wasn't code that didn't get changed from
'openstack.common.xyz' to 'nova.openstack.common.xyz' (hint: this
happens some times)
4. git checkout openstack-common.conf to revert the changes to
openstack-common.conf

IMHO, update.py needs a bit of work (well, I think the whole code
copying thing needs a bit of work, but that's a different story).

Best Regards,
Solly Ross

- Original Message -
From: Jay S Bryant jsbry...@us.ibm.com
To: OpenStack Development Mailing List (not for usage questions)
openstack-dev@lists.openstack.org
Sent: Friday, March 14, 2014 3:36:49 PM
Subject: Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py




From: Brant Knudson b...@acm.org
To: OpenStack Development Mailing List (not for usage questions)
openstack-dev@lists.openstack.org,
Date: 03/14/2014 02:21 PM
Subject: Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py







On Fri, Mar 14, 2014 at 2:05 PM, Jay S Bryant  jsbry...@us.ibm.com  
wrote:


It would be great if we could get the process for this automated. In 
the
mean time, those of us doing the syncs will just have to slog through 
the

process.

Jay



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


What's the process? How do I generate the list of changes?

Brant
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Brant,

My process thus far has been the following:


1. Do the sync to see what files are changed.
2. Take a look at the last commit sync'd to what is currently in
master for a file.
3. Document all the commits that have come in on that file since.
4. Repeat process for all the relevant files if there is more than 
one.

5. If are multiples files I organize the commits with a list of
the files touched by that commit.
6. Document the master level of Oslo when the sync was done for 
reference.


Process may not be perfect, but it gets the job done. Here is an
example of the format I use: https://review.openstack.org/#/c/75740/

Jay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2014-03-12 Thread Duncan Thomas
On 15 January 2014 18:53, Brant Knudson b...@acm.org wrote:

 At no point do I care what are the different commits that are being brought
 in from oslo-incubator. If the commits are listed in the commit message then
 I feel an obligation to verify that they got the right commits in the
 message and that takes extra time for no gain.

I find that I very much *do* want a list of what changes have been
pulled in, so I've so idea of the intent of the changes. Some of the
OSLO changes can be large and complicated, and the more clues as to
why things changed, the better the chance I've got of spotting
breakages or differing assumptions between cinder and OSLO (of which
there have been a number)

I don't very often verify that the version that has been pulled in is
the very latest or anything like that - generally I want to know:
 - What issue are you trying to fix by doing an update? (The fact OSLO
is ahead of us is rarely a good enough reason on its own to do an
update - preferably reference a specific bug that exists in cinder)
 - What other incidental changes are being pulled in (by intent, not
just the code)
 - If I'm unsure about one of the incidental changes, how do I go find
the history for it, with lots of searching (hense the commit ID or the
change ID) - this lets me find bugs, reviews etc

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2014-01-15 Thread Flavio Percoco

On 14/01/14 16:33 -0600, Ben Nemec wrote:

On 2014-01-14 15:26, Doug Hellmann wrote:

   In the release meeting today, Russell proposed that we at least include the
   hash of the HEAD when the merge is done, to indicate how far along the oslo
   changes are. More detail is obviously better.

   So, let's consider this as a new policy. Does anyone foresee issues with

   making this work?




+1 from me. Lets keep it simple. I agree with you on the fact that we
should be focusing more on graduating modules from the incubator.


I don't see a problem with doing that, but I'm not clear on where we're
including the hash.  In the file itself, in a separate file, and/or in the
commit message?

Even if we do no more automation, having the commit hash of the last sync would
be immensely helpful.  Not having to comb through commit logs to figure out
when the last sync happened would be fantastic. :-)


We should keep it in the openstack-modules.conf file and put it in the
commit message as well. IMHO.

FF



-Ben





   On Tue, Jan 14, 2014 at 4:23 AM, Flavio Percoco fla...@redhat.com wrote:

   On 13/01/14 12:07 -0500, Doug Hellmann wrote:
   [.]



   I spent some time trying to think through how we could improve the
   update
   script for [1], and I'm stumped on how to figure out *accurately*
   what state
   the project repositories are in today.

   We can't just compute the hash of the modules in the project
   receiving copies,
   and then look for them in the oslo-incubator repo, because we
   modify the files
   as we copy them out (to update the import statements and replace
   oslo with
   the receiving project name in some places like config option
   defaults).

   We could undo those changes before computing the hash, but the
   problem is
   further complicated because syncs are not being done of all modules
   together.
   The common code in a project doesn't move forward in step with the
   oslo-incubator repository as a whole. For example, sometimes only
   the openstack
   /common/log.py module is copied and not all of openstack/common. So
   log.py
   might be newer than a lot of the rest of the oslo code. The problem
   is even
   worse for something like rpc, where it's possible that modules
   within the rpc
   package might not all be updated together.

   We could probably spend a lot of effort building a tool to tell us
   exactly what
   the state of all of each common file is in each project, to figure
   out what
   needs to be synced. I would much rather spend that effort on
   turning the common
   code into libraries, though.

   So, here's an alternative:

   1. Projects accept a full sync of Oslo soon, including adding a
   value in their
   openstack-common.conf indicating which commit in oslo-incubator is
   reflected in
   the sync. We'll try to make those commit messages as detailed as
   possible.

   2. We modify update.py to remove the option to update individual
   modules when
   copying from oslo-incubator. The new version would always apply all
   changes
   from the last merged commit, as a series of commits, to the
   receiving project.
   So if nova is out of step by 3 commits, then 3 new commits would be
   created in
   the branch by the person doing the update, each with the commit log
   message
   from the change in oslo-incubator. (This lock-step approach is
   necessary to
   have any hope of figuring out which commits are actually being
   synced, so the
   log messages are accurate.)

   In my experience, when syncing files from oslo, it'll most likely
   require syncing more than one module. There's been just *few* times
   where copying a module from oslo resulted in just that specific module
   being copied.

   All that to say that I agree with this point.




   3. The person proposing the merge into the project can decide
   whether to squash
   the commits, or leave them as separate reviews.




   If we use relative imports for modules in oslo incubators (as
   mentioned in another email in this thread) and we *always* keep
   everything up to the latest. What about reconsidering using git
   submodules?

   AFAIR, the main issue with git submodules is that we wanted to support
   updating individual modules. If we remove that option, I think git
   submodules would work just fine. Am I missing something?

   Instead of hacking on update.py we could work on a migration plan out
   of it.

 

Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2014-01-15 Thread Doug Hellmann
On Wed, Jan 15, 2014 at 4:40 AM, Flavio Percoco fla...@redhat.com wrote:

 On 14/01/14 16:33 -0600, Ben Nemec wrote:

 On 2014-01-14 15:26, Doug Hellmann wrote:

In the release meeting today, Russell proposed that we at least
 include the
hash of the HEAD when the merge is done, to indicate how far along the
 oslo
changes are. More detail is obviously better.
So, let's consider this as a new policy. Does anyone foresee
 issues with
making this work?



 +1 from me. Lets keep it simple. I agree with you on the fact that we
 should be focusing more on graduating modules from the incubator.


  I don't see a problem with doing that, but I'm not clear on where we're
 including the hash.  In the file itself, in a separate file, and/or in the
 commit message?

 Even if we do no more automation, having the commit hash of the last sync
 would
 be immensely helpful.  Not having to comb through commit logs to figure
 out
 when the last sync happened would be fantastic. :-)


 We should keep it in the openstack-modules.conf file and put it in the
 commit message as well. IMHO.


I was thinking the commit message, but if you see usefulness in including
the conf file, we could do that, too.

Doug




 FF



 -Ben



On Tue, Jan 14, 2014 at 4:23 AM, Flavio Percoco fla...@redhat.com
 wrote:

On 13/01/14 12:07 -0500, Doug Hellmann wrote:
[.]



I spent some time trying to think through how we could improve
 the
update
script for [1], and I'm stumped on how to figure out
 *accurately*
what state
the project repositories are in today.

We can't just compute the hash of the modules in the project
receiving copies,
and then look for them in the oslo-incubator repo, because we
modify the files
as we copy them out (to update the import statements and
 replace
oslo with
the receiving project name in some places like config option
defaults).

We could undo those changes before computing the hash, but the
problem is
further complicated because syncs are not being done of all
 modules
together.
The common code in a project doesn't move forward in step with
 the
oslo-incubator repository as a whole. For example, sometimes
 only
the openstack
/common/log.py module is copied and not all of
 openstack/common. So
log.py
might be newer than a lot of the rest of the oslo code. The
 problem
is even
worse for something like rpc, where it's possible that modules
within the rpc
package might not all be updated together.

We could probably spend a lot of effort building a tool to
 tell us
exactly what
the state of all of each common file is in each project, to
 figure
out what
needs to be synced. I would much rather spend that effort on
turning the common
code into libraries, though.

So, here's an alternative:

1. Projects accept a full sync of Oslo soon, including adding a
value in their
openstack-common.conf indicating which commit in
 oslo-incubator is
reflected in
the sync. We'll try to make those commit messages as detailed
 as
possible.

2. We modify update.py to remove the option to update
 individual
modules when
copying from oslo-incubator. The new version would always
 apply all
changes
from the last merged commit, as a series of commits, to the
receiving project.
So if nova is out of step by 3 commits, then 3 new commits
 would be
created in
the branch by the person doing the update, each with the
 commit log
message
from the change in oslo-incubator. (This lock-step approach is
necessary to
have any hope of figuring out which commits are actually being
synced, so the
log messages are accurate.)

In my experience, when syncing files from oslo, it'll most likely
require syncing more than one module. There's been just *few* times
where copying a module from oslo resulted in just that specific
 module
being copied.

All that to say that I agree with this point.




3. The person proposing the merge into the project can decide
whether to squash
the commits, or leave them as separate reviews.




If we use relative imports for modules in oslo incubators (as
mentioned in another email in this thread) and we *always* keep
everything up to the latest. What about reconsidering using git
submodules?

AFAIR, 

Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2014-01-15 Thread Flavio Percoco

On 15/01/14 07:44 -0500, Doug Hellmann wrote:
[...]


   I don't see a problem with doing that, but I'm not clear on where we're
   including the hash.  In the file itself, in a separate file, and/or in
   the
   commit message?

   Even if we do no more automation, having the commit hash of the last
   sync would
   be immensely helpful.  Not having to comb through commit logs to figure
   out
   when the last sync happened would be fantastic. :-)


   We should keep it in the openstack-modules.conf file and put it in the
   commit message as well. IMHO.


I was thinking the commit message, but if you see usefulness in including the
conf file, we could do that, too.


As for now, the use I can think of is to know what was the last
oslo-incubator commit that was synced. We could get the same info by
using git log and copying the commit but it's easier to check that
file. (Pretty much like submodules do).

And if we ever need to do something with that, it'll be easier to get
the sha from the config file. This argument violates YAGNI, though... :D

FF

--
@flaper87
Flavio Percoco


pgpg2_2UF3e1S.pgp
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2014-01-15 Thread Ben Nemec

On 2014-01-15 07:05, Flavio Percoco wrote:

On 15/01/14 07:44 -0500, Doug Hellmann wrote:
[...]


   I don't see a problem with doing that, but I'm not clear on 
where we're
   including the hash.  In the file itself, in a separate file, 
and/or in

   the
   commit message?

   Even if we do no more automation, having the commit hash of the 
last

   sync would
   be immensely helpful.  Not having to comb through commit logs 
to figure

   out
   when the last sync happened would be fantastic. :-)


   We should keep it in the openstack-modules.conf file and put it in 
the

   commit message as well. IMHO.


I was thinking the commit message, but if you see usefulness in 
including the

conf file, we could do that, too.


As for now, the use I can think of is to know what was the last
oslo-incubator commit that was synced. We could get the same info by
using git log and copying the commit but it's easier to check that
file. (Pretty much like submodules do).

And if we ever need to do something with that, it'll be easier to get
the sha from the config file. This argument violates YAGNI, though... 
:D


To me, not having to search the target project's git log to find the 
last sync is enough need to justify it.  Storing the last sync hash 
shouldn't be a significant amount of work if we're already going to be 
putting it in the commit hash, and if it makes syncs a little easier 
then I think it's absolutely worth it.


-Ben

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2014-01-15 Thread Brant Knudson
Here's the process I use to review a oslo-incubator merge... since I'm
mostly reviewing keystone I'm going to use that as the project.

1) Make sure keystone master is at the latest, and that oslo-incubator is
at the right level (the commit hash if they mentioned it or latest)
2) run update.py in oslo-incubator, make sure that there's changes in
keystone (maybe we have it already from a different review)
3) revert changes in keystone and check out the review with git-review -d
4) run update.py again and make sure there's no changes in keystone -- If
there's changes then I -1 the review, since I can't verify it was done
correctly.
5) Then I go through the review and look for problems... changes that
aren't backwards compatible, new config options for the sample config file,
etc.

At no point do I care what are the different commits that are being brought
in from oslo-incubator. If the commits are listed in the commit message
then I feel an obligation to verify that they got the right commits in the
message and that takes extra time for no gain.

So I would prefer it if the message for syncs from oslo-incubator included
the commit hash and did not include a list of changes.

If generating the list of changes was automated then I'd add a step to my
process to verify the commit message was the same.

- Brant



On Wed, Jan 15, 2014 at 6:44 AM, Doug Hellmann
doug.hellm...@dreamhost.comwrote:




 On Wed, Jan 15, 2014 at 4:40 AM, Flavio Percoco fla...@redhat.com wrote:

 On 14/01/14 16:33 -0600, Ben Nemec wrote:

 On 2014-01-14 15:26, Doug Hellmann wrote:

In the release meeting today, Russell proposed that we at least
 include the
hash of the HEAD when the merge is done, to indicate how far along
 the oslo
changes are. More detail is obviously better.
So, let's consider this as a new policy. Does anyone foresee
 issues with
making this work?



 +1 from me. Lets keep it simple. I agree with you on the fact that we
 should be focusing more on graduating modules from the incubator.


  I don't see a problem with doing that, but I'm not clear on where we're
 including the hash.  In the file itself, in a separate file, and/or in
 the
 commit message?

 Even if we do no more automation, having the commit hash of the last
 sync would
 be immensely helpful.  Not having to comb through commit logs to figure
 out
 when the last sync happened would be fantastic. :-)


 We should keep it in the openstack-modules.conf file and put it in the
 commit message as well. IMHO.


 I was thinking the commit message, but if you see usefulness in including
 the conf file, we could do that, too.

 Doug




 FF



 -Ben



On Tue, Jan 14, 2014 at 4:23 AM, Flavio Percoco fla...@redhat.com
 wrote:

On 13/01/14 12:07 -0500, Doug Hellmann wrote:
[.]



I spent some time trying to think through how we could
 improve the
update
script for [1], and I'm stumped on how to figure out
 *accurately*
what state
the project repositories are in today.

We can't just compute the hash of the modules in the project
receiving copies,
and then look for them in the oslo-incubator repo, because we
modify the files
as we copy them out (to update the import statements and
 replace
oslo with
the receiving project name in some places like config option
defaults).

We could undo those changes before computing the hash, but the
problem is
further complicated because syncs are not being done of all
 modules
together.
The common code in a project doesn't move forward in step
 with the
oslo-incubator repository as a whole. For example, sometimes
 only
the openstack
/common/log.py module is copied and not all of
 openstack/common. So
log.py
might be newer than a lot of the rest of the oslo code. The
 problem
is even
worse for something like rpc, where it's possible that modules
within the rpc
package might not all be updated together.

We could probably spend a lot of effort building a tool to
 tell us
exactly what
the state of all of each common file is in each project, to
 figure
out what
needs to be synced. I would much rather spend that effort on
turning the common
code into libraries, though.

So, here's an alternative:

1. Projects accept a full sync of Oslo soon, including adding
 a
value in their
openstack-common.conf indicating which commit in
 oslo-incubator is
reflected in
the sync. We'll try to make those commit messages as detailed
 as
possible.

2. We modify update.py to remove the option to update

Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2014-01-15 Thread Doug Hellmann
On Wed, Jan 15, 2014 at 1:53 PM, Brant Knudson b...@acm.org wrote:


 Here's the process I use to review a oslo-incubator merge... since I'm
 mostly reviewing keystone I'm going to use that as the project.

 1) Make sure keystone master is at the latest, and that oslo-incubator is
 at the right level (the commit hash if they mentioned it or latest)
 2) run update.py in oslo-incubator, make sure that there's changes in
 keystone (maybe we have it already from a different review)
 3) revert changes in keystone and check out the review with git-review -d
 4) run update.py again and make sure there's no changes in keystone -- If
 there's changes then I -1 the review, since I can't verify it was done
 correctly.
 5) Then I go through the review and look for problems... changes that
 aren't backwards compatible, new config options for the sample config file,
 etc.


The first few steps of that seem like something that could be automated
with a fairly simple script.


 At no point do I care what are the different commits that are being
 brought in from oslo-incubator. If the commits are listed in the commit
 message then I feel an obligation to verify that they got the right commits
 in the message and that takes extra time for no gain.


That makes sense to me. The request from the summit was to include the
commits, but I think so far everyone is comfortable with backing that off
to just say the most recent hash.

Doug




 So I would prefer it if the message for syncs from oslo-incubator included
 the commit hash and did not include a list of changes.

 If generating the list of changes was automated then I'd add a step to my
 process to verify the commit message was the same.

 - Brant



 On Wed, Jan 15, 2014 at 6:44 AM, Doug Hellmann 
 doug.hellm...@dreamhost.com wrote:




 On Wed, Jan 15, 2014 at 4:40 AM, Flavio Percoco fla...@redhat.comwrote:

 On 14/01/14 16:33 -0600, Ben Nemec wrote:

 On 2014-01-14 15:26, Doug Hellmann wrote:

In the release meeting today, Russell proposed that we at least
 include the
hash of the HEAD when the merge is done, to indicate how far along
 the oslo
changes are. More detail is obviously better.
So, let's consider this as a new policy. Does anyone foresee
 issues with
making this work?



 +1 from me. Lets keep it simple. I agree with you on the fact that we
 should be focusing more on graduating modules from the incubator.


  I don't see a problem with doing that, but I'm not clear on where we're
 including the hash.  In the file itself, in a separate file, and/or in
 the
 commit message?

 Even if we do no more automation, having the commit hash of the last
 sync would
 be immensely helpful.  Not having to comb through commit logs to figure
 out
 when the last sync happened would be fantastic. :-)


 We should keep it in the openstack-modules.conf file and put it in the
 commit message as well. IMHO.


 I was thinking the commit message, but if you see usefulness in including
 the conf file, we could do that, too.

 Doug




 FF



 -Ben



On Tue, Jan 14, 2014 at 4:23 AM, Flavio Percoco fla...@redhat.com
 wrote:

On 13/01/14 12:07 -0500, Doug Hellmann wrote:
[.]



I spent some time trying to think through how we could
 improve the
update
script for [1], and I'm stumped on how to figure out
 *accurately*
what state
the project repositories are in today.

We can't just compute the hash of the modules in the project
receiving copies,
and then look for them in the oslo-incubator repo, because we
modify the files
as we copy them out (to update the import statements and
 replace
oslo with
the receiving project name in some places like config option
defaults).

We could undo those changes before computing the hash, but
 the
problem is
further complicated because syncs are not being done of all
 modules
together.
The common code in a project doesn't move forward in step
 with the
oslo-incubator repository as a whole. For example, sometimes
 only
the openstack
/common/log.py module is copied and not all of
 openstack/common. So
log.py
might be newer than a lot of the rest of the oslo code. The
 problem
is even
worse for something like rpc, where it's possible that
 modules
within the rpc
package might not all be updated together.

We could probably spend a lot of effort building a tool to
 tell us
exactly what
the state of all of each common file is in each project, to
 figure
out what
needs to be synced. I would much rather spend that effort on
turning the common
code into libraries, though.

So, here's an 

Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2014-01-14 Thread Flavio Percoco

On 13/01/14 12:07 -0500, Doug Hellmann wrote:
[.]


I spent some time trying to think through how we could improve the update
script for [1], and I'm stumped on how to figure out *accurately* what state
the project repositories are in today.

We can't just compute the hash of the modules in the project receiving copies,
and then look for them in the oslo-incubator repo, because we modify the files
as we copy them out (to update the import statements and replace oslo with
the receiving project name in some places like config option defaults).

We could undo those changes before computing the hash, but the problem is
further complicated because syncs are not being done of all modules together.
The common code in a project doesn't move forward in step with the
oslo-incubator repository as a whole. For example, sometimes only the openstack
/common/log.py module is copied and not all of openstack/common. So log.py
might be newer than a lot of the rest of the oslo code. The problem is even
worse for something like rpc, where it's possible that modules within the rpc
package might not all be updated together.

We could probably spend a lot of effort building a tool to tell us exactly what
the state of all of each common file is in each project, to figure out what
needs to be synced. I would much rather spend that effort on turning the common
code into libraries, though.

So, here's an alternative:

1. Projects accept a full sync of Oslo soon, including adding a value in their
openstack-common.conf indicating which commit in oslo-incubator is reflected in
the sync. We'll try to make those commit messages as detailed as possible.

2. We modify update.py to remove the option to update individual modules when
copying from oslo-incubator. The new version would always apply all changes
from the last merged commit, as a series of commits, to the receiving project.
So if nova is out of step by 3 commits, then 3 new commits would be created in
the branch by the person doing the update, each with the commit log message
from the change in oslo-incubator. (This lock-step approach is necessary to
have any hope of figuring out which commits are actually being synced, so the
log messages are accurate.)


In my experience, when syncing files from oslo, it'll most likely
require syncing more than one module. There's been just *few* times
where copying a module from oslo resulted in just that specific module
being copied.

All that to say that I agree with this point.



3. The person proposing the merge into the project can decide whether to squash
the commits, or leave them as separate reviews.




If we use relative imports for modules in oslo incubators (as
mentioned in another email in this thread) and we *always* keep
everything up to the latest. What about reconsidering using git
submodules?

AFAIR, the main issue with git submodules is that we wanted to support
updating individual modules. If we remove that option, I think git
submodules would work just fine. Am I missing something?

Instead of hacking on update.py we could work on a migration plan out
of it.

A downside of using submodules is that when moving the reference in
the submodule, it won't be obvious why that's happening, which is
something we wanted to fix with update.py. It would be up to the
committer to write a good commit message or to get the messages out of
the submodule history.

Another downside is that it would be hard to apply isolated patches on
stable branches for security issues or really awful bugs.

I'm less convinced about submodules now but I'm leaving this in the
email in case someone wants to dig a bit deeper in the topic.


I'm not entirely certain I like this approach myself, but it's the best I've
been able to come up with. It essentially gives us the current process, while
removing the ability to potentially take a version of a module without taking
its dependencies (allowing us to step forward, and track the commit messages
accurately). It will also produce results similar to what we will have when all
of this oslo code moves into separate libraries, where the changes to the
library will be seen by the projects without any action at all on their part.


After going through this for a bit, I agree with you. The goal
of the update script should be:

   - Sync modules from the current state to the most updated version

   - Make sure the update information is not lost. If there's an oslo
 sync review without the commits shas + description, it simply
 means the committer amended the message.


OTOH, it will also require spending time on update.py, instead of releasing a
library from the incubator. And it doesn't really buy us that much in terms of
making the sync happen more easily, other than a reliable way of having
entirely accurate commit messages.


Although it distracts us from our real goal - releasing libraries - I
still think is necessary. We should probably just reduce the changes
needed as much as 

Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2014-01-14 Thread Doug Hellmann
In the release meeting today, Russell proposed that we at least include the
hash of the HEAD when the merge is done, to indicate how far along the oslo
changes are. More detail is obviously better.

So, let's consider this as a new policy. Does anyone foresee issues with
making this work?

Doug



On Tue, Jan 14, 2014 at 4:23 AM, Flavio Percoco fla...@redhat.com wrote:

 On 13/01/14 12:07 -0500, Doug Hellmann wrote:
 [.]


 I spent some time trying to think through how we could improve the update
 script for [1], and I'm stumped on how to figure out *accurately* what
 state
 the project repositories are in today.

 We can't just compute the hash of the modules in the project receiving
 copies,
 and then look for them in the oslo-incubator repo, because we modify the
 files
 as we copy them out (to update the import statements and replace oslo
 with
 the receiving project name in some places like config option defaults).

 We could undo those changes before computing the hash, but the problem is
 further complicated because syncs are not being done of all modules
 together.
 The common code in a project doesn't move forward in step with the
 oslo-incubator repository as a whole. For example, sometimes only the
 openstack
 /common/log.py module is copied and not all of openstack/common. So log.py
 might be newer than a lot of the rest of the oslo code. The problem is
 even
 worse for something like rpc, where it's possible that modules within the
 rpc
 package might not all be updated together.

 We could probably spend a lot of effort building a tool to tell us
 exactly what
 the state of all of each common file is in each project, to figure out
 what
 needs to be synced. I would much rather spend that effort on turning the
 common
 code into libraries, though.

 So, here's an alternative:

 1. Projects accept a full sync of Oslo soon, including adding a value in
 their
 openstack-common.conf indicating which commit in oslo-incubator is
 reflected in
 the sync. We'll try to make those commit messages as detailed as possible.

 2. We modify update.py to remove the option to update individual modules
 when
 copying from oslo-incubator. The new version would always apply all
 changes
 from the last merged commit, as a series of commits, to the receiving
 project.
 So if nova is out of step by 3 commits, then 3 new commits would be
 created in
 the branch by the person doing the update, each with the commit log
 message
 from the change in oslo-incubator. (This lock-step approach is necessary
 to
 have any hope of figuring out which commits are actually being synced, so
 the
 log messages are accurate.)


 In my experience, when syncing files from oslo, it'll most likely
 require syncing more than one module. There's been just *few* times
 where copying a module from oslo resulted in just that specific module
 being copied.

 All that to say that I agree with this point.



 3. The person proposing the merge into the project can decide whether to
 squash
 the commits, or leave them as separate reviews.



 If we use relative imports for modules in oslo incubators (as
 mentioned in another email in this thread) and we *always* keep
 everything up to the latest. What about reconsidering using git
 submodules?

 AFAIR, the main issue with git submodules is that we wanted to support
 updating individual modules. If we remove that option, I think git
 submodules would work just fine. Am I missing something?

 Instead of hacking on update.py we could work on a migration plan out
 of it.

 A downside of using submodules is that when moving the reference in
 the submodule, it won't be obvious why that's happening, which is
 something we wanted to fix with update.py. It would be up to the
 committer to write a good commit message or to get the messages out of
 the submodule history.

 Another downside is that it would be hard to apply isolated patches on
 stable branches for security issues or really awful bugs.

 I'm less convinced about submodules now but I'm leaving this in the
 email in case someone wants to dig a bit deeper in the topic.


  I'm not entirely certain I like this approach myself, but it's the best
 I've
 been able to come up with. It essentially gives us the current process,
 while
 removing the ability to potentially take a version of a module without
 taking
 its dependencies (allowing us to step forward, and track the commit
 messages
 accurately). It will also produce results similar to what we will have
 when all
 of this oslo code moves into separate libraries, where the changes to the
 library will be seen by the projects without any action at all on their
 part.


 After going through this for a bit, I agree with you. The goal
 of the update script should be:

- Sync modules from the current state to the most updated version

- Make sure the update information is not lost. If there's an oslo
  sync review without the commits shas + description, it simply
  

Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2014-01-14 Thread Ben Nemec
 

On 2014-01-14 15:26, Doug Hellmann wrote: 

 In the release meeting today, Russell proposed that we at least include the 
 hash of the HEAD when the merge is done, to indicate how far along the oslo 
 changes are. More detail is obviously better. 
 
 So, let's consider this as a new policy. Does anyone foresee issues with 
 making this work? 
 
 Doug

I don't see a problem with doing that, but I'm not clear on where we're
including the hash. In the file itself, in a separate file, and/or in
the commit message? 

Even if we do no more automation, having the commit hash of the last
sync would be immensely helpful. Not having to comb through commit logs
to figure out when the last sync happened would be fantastic. :-) 

-Ben 

 On Tue, Jan 14, 2014 at 4:23 AM, Flavio Percoco fla...@redhat.com wrote:
 On 13/01/14 12:07 -0500, Doug Hellmann wrote:
 [.] 
 
 I spent some time trying to think through how we could improve the update
 script for [1], and I'm stumped on how to figure out *accurately* what state
 the project repositories are in today.
 
 We can't just compute the hash of the modules in the project receiving copies,
 and then look for them in the oslo-incubator repo, because we modify the files
 as we copy them out (to update the import statements and replace oslo with
 the receiving project name in some places like config option defaults).
 
 We could undo those changes before computing the hash, but the problem is
 further complicated because syncs are not being done of all modules together.
 The common code in a project doesn't move forward in step with the
 oslo-incubator repository as a whole. For example, sometimes only the 
 openstack
 /common/log.py module is copied and not all of openstack/common. So log.py
 might be newer than a lot of the rest of the oslo code. The problem is even
 worse for something like rpc, where it's possible that modules within the rpc
 package might not all be updated together.
 
 We could probably spend a lot of effort building a tool to tell us exactly 
 what
 the state of all of each common file is in each project, to figure out what
 needs to be synced. I would much rather spend that effort on turning the 
 common
 code into libraries, though.
 
 So, here's an alternative:
 
 1. Projects accept a full sync of Oslo soon, including adding a value in their
 openstack-common.conf indicating which commit in oslo-incubator is reflected 
 in
 the sync. We'll try to make those commit messages as detailed as possible.
 
 2. We modify update.py to remove the option to update individual modules when
 copying from oslo-incubator. The new version would always apply all changes
 from the last merged commit, as a series of commits, to the receiving project.
 So if nova is out of step by 3 commits, then 3 new commits would be created in
 the branch by the person doing the update, each with the commit log message
 from the change in oslo-incubator. (This lock-step approach is necessary to
 have any hope of figuring out which commits are actually being synced, so the
 log messages are accurate.) In my experience, when syncing files from oslo, 
 it'll most likely
 require syncing more than one module. There's been just *few* times
 where copying a module from oslo resulted in just that specific module
 being copied.
 
 All that to say that I agree with this point. 
 
 3. The person proposing the merge into the project can decide whether to 
 squash
 the commits, or leave them as separate reviews.
 
 If we use relative imports for modules in oslo incubators (as
 mentioned in another email in this thread) and we *always* keep
 everything up to the latest. What about reconsidering using git
 submodules?
 
 AFAIR, the main issue with git submodules is that we wanted to support
 updating individual modules. If we remove that option, I think git
 submodules would work just fine. Am I missing something?
 
 Instead of hacking on update.py we could work on a migration plan out
 of it.
 
 A downside of using submodules is that when moving the reference in
 the submodule, it won't be obvious why that's happening, which is
 something we wanted to fix with update.py. It would be up to the
 committer to write a good commit message or to get the messages out of
 the submodule history.
 
 Another downside is that it would be hard to apply isolated patches on
 stable branches for security issues or really awful bugs.
 
 I'm less convinced about submodules now but I'm leaving this in the
 email in case someone wants to dig a bit deeper in the topic. 
 
 I'm not entirely certain I like this approach myself, but it's the best I've
 been able to come up with. It essentially gives us the current process, while
 removing the ability to potentially take a version of a module without taking
 its dependencies (allowing us to step forward, and track the commit messages
 accurately). It will also produce results similar to what we will have when 
 all
 of this oslo code moves into separate 

Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2014-01-13 Thread Doug Hellmann
[resurrecting an old thread]


On Wed, Nov 27, 2013 at 6:26 AM, Flavio Percoco fla...@redhat.com wrote:

 On 27/11/13 10:59 +, Mark McLoughlin wrote:

 On Wed, 2013-11-27 at 11:50 +0100, Flavio Percoco wrote:

 On 26/11/13 22:54 +, Mark McLoughlin wrote:
 On Fri, 2013-11-22 at 12:39 -0500, Doug Hellmann wrote:
  On Fri, Nov 22, 2013 at 4:11 AM, Flavio Percoco fla...@redhat.com
 wrote:
  1) Store the commit sha from which the module was copied from.
  Every project using oslo, currently keeps the list of modules it
  is using in `openstack-modules.conf` in a `module` parameter. We
  could store, along with the module name, the sha of the commit it
  was last synced from:
  
  module=log,commit
  
  or
  
  module=log
  log=commit
  
 
  The second form will be easier to manage. Humans edit the module
 field and
  the script will edit the others.
 
 How about adding it as a comment at the end of the python files
 themselves and leaving openstack-common.conf for human editing?

 I think having the commit sha will give us a starting point from which
 we could start updating that module from.


 Sure, my only point was about where the commit sha comes from - i.e.
 whether it's from a comment at the end of the python module itself or in
 openstack-common.conf


 And, indeed you said 'at the end of the python files'. Don't ask me
 how the heck I misread that.

 The benefit I see from having them in the openstack-common.conf is
 that we can register a `StrOpt` for each object dynamically and get
 the sha using oslo.config. If we put it as a comment at the end of the
 python file, we'll have to read it and 'parse' it, I guess.



  It will mostly help with
 getting a diff for that module and the short commit messages where it
 was modified.

 Here's a pseudo-buggy-algorithm for the update process:

 (1) Get current sha for $module
 (2) Get list of new commits for $module
 (3) for each commit of $module:
 (3.1) for each modified_module in $commit
 (3.1.1) Update those modules up to $commit
 (1)(modified_module)
 (3.2) Copy the new file
 (3.3) Update openstack-common with the latest sha

 This trusts the granularity and isolation of the patches proposed in
 oslo-incubator. However, in cases like 'remove vim mode lines' it'll
 fail assuming that updating every module is necessary - which is true
 from a git stand point.


 This is another variant of the kind of inter-module dependency smarts
 that update.py already has ... I'd be inclined to just omit those smarts
 and just require the caller to explicitly list the modules they want to
 include.

 Maybe update.py could include some reporting to help with that choice
 like module foo depends on modules bar and blaa, maybe you want to
 include them too and commit XXX modified module foo, but also module
 bar and blaa, maybe you want to include them too.


 But, if we get to the point of suggesting the user to update module
 foo because it was modified in commit XXX, we'd have everything needed
 to make it recursive and update those modules as well.

 I agree with you on making it explicit, though. What about making it
 interactive then? update.py could ask users if they want to update
 module foo because it was modified in commit XXX and do it right away,
 which is not very different from updating module foo, print a report
 and let the user choose afterwards.

 (/me feels like Gollum now)

 I prefer the interactive way though, at least it doesn't require the
 user to run update several times for each module. We could also add a
 `--no-stop` flag that does exactly what you suggested.


I spent some time trying to think through how we could improve the update
script for [1], and I'm stumped on how to figure out *accurately* what
state the project repositories are in today.

We can't just compute the hash of the modules in the project receiving
copies, and then look for them in the oslo-incubator repo, because we
modify the files as we copy them out (to update the import statements and
replace oslo with the receiving project name in some places like config
option defaults).

We could undo those changes before computing the hash, but the problem is
further complicated because syncs are not being done of all modules
together. The common code in a project doesn't move forward in step with
the oslo-incubator repository as a whole. For example, sometimes only the
openstack/common/log.py module is copied and not all of openstack/common.
So log.py might be newer than a lot of the rest of the oslo code. The
problem is even worse for something like rpc, where it's possible that
modules within the rpc package might not all be updated together.

We could probably spend a lot of effort building a tool to tell us exactly
what the state of all of each common file is in each project, to figure out
what needs to be synced. I would much rather spend that effort on turning
the 

Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2014-01-13 Thread Robert Myers
We could always use relative imports in oslo :) Then you could put it where
ever you wanted to without needing to rewrite the import statements.


On Mon, Jan 13, 2014 at 11:07 AM, Doug Hellmann doug.hellm...@dreamhost.com
 wrote:

 [resurrecting an old thread]


 On Wed, Nov 27, 2013 at 6:26 AM, Flavio Percoco fla...@redhat.com wrote:

 On 27/11/13 10:59 +, Mark McLoughlin wrote:

 On Wed, 2013-11-27 at 11:50 +0100, Flavio Percoco wrote:

 On 26/11/13 22:54 +, Mark McLoughlin wrote:
 On Fri, 2013-11-22 at 12:39 -0500, Doug Hellmann wrote:
  On Fri, Nov 22, 2013 at 4:11 AM, Flavio Percoco fla...@redhat.com
 wrote:
  1) Store the commit sha from which the module was copied from.
  Every project using oslo, currently keeps the list of modules it
  is using in `openstack-modules.conf` in a `module` parameter. We
  could store, along with the module name, the sha of the commit
 it
  was last synced from:
  
  module=log,commit
  
  or
  
  module=log
  log=commit
  
 
  The second form will be easier to manage. Humans edit the module
 field and
  the script will edit the others.
 
 How about adding it as a comment at the end of the python files
 themselves and leaving openstack-common.conf for human editing?

 I think having the commit sha will give us a starting point from which
 we could start updating that module from.


 Sure, my only point was about where the commit sha comes from - i.e.
 whether it's from a comment at the end of the python module itself or in
 openstack-common.conf


 And, indeed you said 'at the end of the python files'. Don't ask me
 how the heck I misread that.

 The benefit I see from having them in the openstack-common.conf is
 that we can register a `StrOpt` for each object dynamically and get
 the sha using oslo.config. If we put it as a comment at the end of the
 python file, we'll have to read it and 'parse' it, I guess.



  It will mostly help with
 getting a diff for that module and the short commit messages where it
 was modified.

 Here's a pseudo-buggy-algorithm for the update process:

 (1) Get current sha for $module
 (2) Get list of new commits for $module
 (3) for each commit of $module:
 (3.1) for each modified_module in $commit
 (3.1.1) Update those modules up to $commit
 (1)(modified_module)
 (3.2) Copy the new file
 (3.3) Update openstack-common with the latest sha

 This trusts the granularity and isolation of the patches proposed in
 oslo-incubator. However, in cases like 'remove vim mode lines' it'll
 fail assuming that updating every module is necessary - which is true
 from a git stand point.


 This is another variant of the kind of inter-module dependency smarts
 that update.py already has ... I'd be inclined to just omit those smarts
 and just require the caller to explicitly list the modules they want to
 include.

 Maybe update.py could include some reporting to help with that choice
 like module foo depends on modules bar and blaa, maybe you want to
 include them too and commit XXX modified module foo, but also module
 bar and blaa, maybe you want to include them too.


 But, if we get to the point of suggesting the user to update module
 foo because it was modified in commit XXX, we'd have everything needed
 to make it recursive and update those modules as well.

 I agree with you on making it explicit, though. What about making it
 interactive then? update.py could ask users if they want to update
 module foo because it was modified in commit XXX and do it right away,
 which is not very different from updating module foo, print a report
 and let the user choose afterwards.

 (/me feels like Gollum now)

 I prefer the interactive way though, at least it doesn't require the
 user to run update several times for each module. We could also add a
 `--no-stop` flag that does exactly what you suggested.


 I spent some time trying to think through how we could improve the update
 script for [1], and I'm stumped on how to figure out *accurately* what
 state the project repositories are in today.

 We can't just compute the hash of the modules in the project receiving
 copies, and then look for them in the oslo-incubator repo, because we
 modify the files as we copy them out (to update the import statements and
 replace oslo with the receiving project name in some places like config
 option defaults).

 We could undo those changes before computing the hash, but the problem is
 further complicated because syncs are not being done of all modules
 together. The common code in a project doesn't move forward in step with
 the oslo-incubator repository as a whole. For example, sometimes only the
 openstack/common/log.py module is copied and not all of openstack/common.
 So log.py might be newer than a lot of the rest of the oslo code. The
 problem is even worse for something like rpc, where it's possible that
 modules within the rpc package might not all be 

Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2014-01-13 Thread Doug Hellmann
On Mon, Jan 13, 2014 at 1:16 PM, Robert Myers myer0...@gmail.com wrote:

 We could always use relative imports in oslo :) Then you could put it
 where ever you wanted to without needing to rewrite the import statements.


That may be a good idea, but doesn't really solve the problem at hand.

Doug





 On Mon, Jan 13, 2014 at 11:07 AM, Doug Hellmann 
 doug.hellm...@dreamhost.com wrote:

 [resurrecting an old thread]


 On Wed, Nov 27, 2013 at 6:26 AM, Flavio Percoco fla...@redhat.comwrote:

 On 27/11/13 10:59 +, Mark McLoughlin wrote:

 On Wed, 2013-11-27 at 11:50 +0100, Flavio Percoco wrote:

 On 26/11/13 22:54 +, Mark McLoughlin wrote:
 On Fri, 2013-11-22 at 12:39 -0500, Doug Hellmann wrote:
  On Fri, Nov 22, 2013 at 4:11 AM, Flavio Percoco fla...@redhat.com
 wrote:
  1) Store the commit sha from which the module was copied from.
  Every project using oslo, currently keeps the list of modules
 it
  is using in `openstack-modules.conf` in a `module` parameter.
 We
  could store, along with the module name, the sha of the commit
 it
  was last synced from:
  
  module=log,commit
  
  or
  
  module=log
  log=commit
  
 
  The second form will be easier to manage. Humans edit the module
 field and
  the script will edit the others.
 
 How about adding it as a comment at the end of the python files
 themselves and leaving openstack-common.conf for human editing?

 I think having the commit sha will give us a starting point from which
 we could start updating that module from.


 Sure, my only point was about where the commit sha comes from - i.e.
 whether it's from a comment at the end of the python module itself or in
 openstack-common.conf


 And, indeed you said 'at the end of the python files'. Don't ask me
 how the heck I misread that.

 The benefit I see from having them in the openstack-common.conf is
 that we can register a `StrOpt` for each object dynamically and get
 the sha using oslo.config. If we put it as a comment at the end of the
 python file, we'll have to read it and 'parse' it, I guess.



  It will mostly help with
 getting a diff for that module and the short commit messages where it
 was modified.

 Here's a pseudo-buggy-algorithm for the update process:

 (1) Get current sha for $module
 (2) Get list of new commits for $module
 (3) for each commit of $module:
 (3.1) for each modified_module in $commit
 (3.1.1) Update those modules up to $commit
 (1)(modified_module)
 (3.2) Copy the new file
 (3.3) Update openstack-common with the latest sha

 This trusts the granularity and isolation of the patches proposed in
 oslo-incubator. However, in cases like 'remove vim mode lines' it'll
 fail assuming that updating every module is necessary - which is true
 from a git stand point.


 This is another variant of the kind of inter-module dependency smarts
 that update.py already has ... I'd be inclined to just omit those smarts
 and just require the caller to explicitly list the modules they want to
 include.

 Maybe update.py could include some reporting to help with that choice
 like module foo depends on modules bar and blaa, maybe you want to
 include them too and commit XXX modified module foo, but also module
 bar and blaa, maybe you want to include them too.


 But, if we get to the point of suggesting the user to update module
 foo because it was modified in commit XXX, we'd have everything needed
 to make it recursive and update those modules as well.

 I agree with you on making it explicit, though. What about making it
 interactive then? update.py could ask users if they want to update
 module foo because it was modified in commit XXX and do it right away,
 which is not very different from updating module foo, print a report
 and let the user choose afterwards.

 (/me feels like Gollum now)

 I prefer the interactive way though, at least it doesn't require the
 user to run update several times for each module. We could also add a
 `--no-stop` flag that does exactly what you suggested.


 I spent some time trying to think through how we could improve the update
 script for [1], and I'm stumped on how to figure out *accurately* what
 state the project repositories are in today.

 We can't just compute the hash of the modules in the project receiving
 copies, and then look for them in the oslo-incubator repo, because we
 modify the files as we copy them out (to update the import statements and
 replace oslo with the receiving project name in some places like config
 option defaults).

 We could undo those changes before computing the hash, but the problem is
 further complicated because syncs are not being done of all modules
 together. The common code in a project doesn't move forward in step with
 the oslo-incubator repository as a whole. For example, sometimes only the
 openstack/common/log.py module is copied and not all of openstack/common.
 So log.py might be newer than a 

Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2014-01-13 Thread Kevin L. Mitchell
On Mon, 2014-01-13 at 12:07 -0500, Doug Hellmann wrote:
 We can't just compute the hash of the modules in the project receiving
 copies, and then look for them in the oslo-incubator repo, because we
 modify the files as we copy them out (to update the import statements
 and replace oslo with the receiving project name in some places like
 config option defaults). 

Why not embed the hash into the files as we copy them out, maybe as a
specially formatted comment at the end of the file?

That said, I've always thought that oslo-incubator should be a library
we use directly, rather than a repository we copy out of.
-- 
Kevin L. Mitchell kevin.mitch...@rackspace.com
Rackspace


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2014-01-13 Thread Doug Hellmann
On Mon, Jan 13, 2014 at 3:41 PM, Kevin L. Mitchell 
kevin.mitch...@rackspace.com wrote:

 On Mon, 2014-01-13 at 12:07 -0500, Doug Hellmann wrote:
  We can't just compute the hash of the modules in the project receiving
  copies, and then look for them in the oslo-incubator repo, because we
  modify the files as we copy them out (to update the import statements
  and replace oslo with the receiving project name in some places like
  config option defaults).

 Why not embed the hash into the files as we copy them out, maybe as a
 specially formatted comment at the end of the file?


The problem is much much easier for the second update. The real challenge
is getting the first update right, when we don't know what the initial hash
is.




 That said, I've always thought that oslo-incubator should be a library
 we use directly, rather than a repository we copy out of.


We're working on that. That's another reason I don't really want to spend
a lot of time making update.py fancier -- it takes away from time I could
be spending on making the code into libraries in the first place.

Doug



 --
 Kevin L. Mitchell kevin.mitch...@rackspace.com
 Rackspace


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2013-11-27 Thread Flavio Percoco

On 26/11/13 22:54 +, Mark McLoughlin wrote:

On Fri, 2013-11-22 at 12:39 -0500, Doug Hellmann wrote:

On Fri, Nov 22, 2013 at 4:11 AM, Flavio Percoco fla...@redhat.com wrote:
1) Store the commit sha from which the module was copied from.
Every project using oslo, currently keeps the list of modules it
is using in `openstack-modules.conf` in a `module` parameter. We
could store, along with the module name, the sha of the commit it
was last synced from:

module=log,commit

or

module=log
log=commit


The second form will be easier to manage. Humans edit the module field and
the script will edit the others.


How about adding it as a comment at the end of the python files
themselves and leaving openstack-common.conf for human editing?


I think having the commit sha will give us a starting point from which
we could start updating that module from. It will mostly help with
getting a diff for that module and the short commit messages where it
was modified.

Here's a pseudo-buggy-algorithm for the update process:

   (1) Get current sha for $module
   (2) Get list of new commits for $module
   (3) for each commit of $module:
   (3.1) for each modified_module in $commit
   (3.1.1) Update those modules up to $commit (1)(modified_module)
   (3.2) Copy the new file
   (3.3) Update openstack-common with the latest sha

This trusts the granularity and isolation of the patches proposed in
oslo-incubator. However, in cases like 'remove vim mode lines' it'll
fail assuming that updating every module is necessary - which is true
from a git stand point.

The above will make update.py way smarter - and complex - than it's
today, but given the number of projects - new and existing - we have
now, I think it's worth it.

Thoughts?

FF

--
@flaper87
Flavio Percoco


pgptFgZslWEs0.pgp
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2013-11-27 Thread Mark McLoughlin
On Wed, 2013-11-27 at 11:50 +0100, Flavio Percoco wrote:
 On 26/11/13 22:54 +, Mark McLoughlin wrote:
 On Fri, 2013-11-22 at 12:39 -0500, Doug Hellmann wrote:
  On Fri, Nov 22, 2013 at 4:11 AM, Flavio Percoco fla...@redhat.com wrote:
  1) Store the commit sha from which the module was copied from.
  Every project using oslo, currently keeps the list of modules it
  is using in `openstack-modules.conf` in a `module` parameter. We
  could store, along with the module name, the sha of the commit it
  was last synced from:
  
  module=log,commit
  
  or
  
  module=log
  log=commit
  
 
  The second form will be easier to manage. Humans edit the module field and
  the script will edit the others.
 
 How about adding it as a comment at the end of the python files
 themselves and leaving openstack-common.conf for human editing?
 
 I think having the commit sha will give us a starting point from which
 we could start updating that module from. 

Sure, my only point was about where the commit sha comes from - i.e.
whether it's from a comment at the end of the python module itself or in
openstack-common.conf

 It will mostly help with
 getting a diff for that module and the short commit messages where it
 was modified.
 
 Here's a pseudo-buggy-algorithm for the update process:
 
 (1) Get current sha for $module
 (2) Get list of new commits for $module
 (3) for each commit of $module:
 (3.1) for each modified_module in $commit
 (3.1.1) Update those modules up to $commit (1)(modified_module)
 (3.2) Copy the new file
 (3.3) Update openstack-common with the latest sha
 
 This trusts the granularity and isolation of the patches proposed in
 oslo-incubator. However, in cases like 'remove vim mode lines' it'll
 fail assuming that updating every module is necessary - which is true
 from a git stand point.

This is another variant of the kind of inter-module dependency smarts
that update.py already has ... I'd be inclined to just omit those smarts
and just require the caller to explicitly list the modules they want to
include.

Maybe update.py could include some reporting to help with that choice
like module foo depends on modules bar and blaa, maybe you want to
include them too and commit XXX modified module foo, but also module
bar and blaa, maybe you want to include them too.

Mark.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2013-11-27 Thread Flavio Percoco

On 27/11/13 10:59 +, Mark McLoughlin wrote:

On Wed, 2013-11-27 at 11:50 +0100, Flavio Percoco wrote:

On 26/11/13 22:54 +, Mark McLoughlin wrote:
On Fri, 2013-11-22 at 12:39 -0500, Doug Hellmann wrote:
 On Fri, Nov 22, 2013 at 4:11 AM, Flavio Percoco fla...@redhat.com wrote:
 1) Store the commit sha from which the module was copied from.
 Every project using oslo, currently keeps the list of modules it
 is using in `openstack-modules.conf` in a `module` parameter. We
 could store, along with the module name, the sha of the commit it
 was last synced from:
 
 module=log,commit
 
 or
 
 module=log
 log=commit
 

 The second form will be easier to manage. Humans edit the module field and
 the script will edit the others.

How about adding it as a comment at the end of the python files
themselves and leaving openstack-common.conf for human editing?

I think having the commit sha will give us a starting point from which
we could start updating that module from.


Sure, my only point was about where the commit sha comes from - i.e.
whether it's from a comment at the end of the python module itself or in
openstack-common.conf


And, indeed you said 'at the end of the python files'. Don't ask me
how the heck I misread that.

The benefit I see from having them in the openstack-common.conf is
that we can register a `StrOpt` for each object dynamically and get
the sha using oslo.config. If we put it as a comment at the end of the
python file, we'll have to read it and 'parse' it, I guess.




It will mostly help with
getting a diff for that module and the short commit messages where it
was modified.

Here's a pseudo-buggy-algorithm for the update process:

(1) Get current sha for $module
(2) Get list of new commits for $module
(3) for each commit of $module:
(3.1) for each modified_module in $commit
(3.1.1) Update those modules up to $commit (1)(modified_module)
(3.2) Copy the new file
(3.3) Update openstack-common with the latest sha

This trusts the granularity and isolation of the patches proposed in
oslo-incubator. However, in cases like 'remove vim mode lines' it'll
fail assuming that updating every module is necessary - which is true
from a git stand point.


This is another variant of the kind of inter-module dependency smarts
that update.py already has ... I'd be inclined to just omit those smarts
and just require the caller to explicitly list the modules they want to
include.

Maybe update.py could include some reporting to help with that choice
like module foo depends on modules bar and blaa, maybe you want to
include them too and commit XXX modified module foo, but also module
bar and blaa, maybe you want to include them too.


But, if we get to the point of suggesting the user to update module
foo because it was modified in commit XXX, we'd have everything needed
to make it recursive and update those modules as well.

I agree with you on making it explicit, though. What about making it
interactive then? update.py could ask users if they want to update
module foo because it was modified in commit XXX and do it right away,
which is not very different from updating module foo, print a report
and let the user choose afterwards.

(/me feels like Gollum now)

I prefer the interactive way though, at least it doesn't require the
user to run update several times for each module. We could also add a
`--no-stop` flag that does exactly what you suggested.

Cheers,
FF

--
@flaper87
Flavio Percoco


pgpjWZxq5ji2s.pgp
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2013-11-26 Thread Yuriy Taraday
Hello.


On Fri, Nov 22, 2013 at 1:11 PM, Flavio Percoco fla...@redhat.com wrote:

1) Store the commit sha from which the module was copied from.


I would suggest we don't duplicate this in every project's repo. It is
possible to find using the contents of the modules currently in project's
repo the corresponding commit from oslo-incubator (see [1] for a hint, feel
free to ask me about Git details). So there's no need to store this
information in a separate file.

   2) Add an 'auto-commit' parameter to the update script that will
generate a commit message with the short log of the commits where
the modules being updated were modified. Soemthing like:

Syncing oslo-incubator modules

log.py:
commit1: short-message
commit2: short-message
commit3: short-message

lockutils:
commit4: short-message
commit5: short-message


The script can use this information as well to get the last synced commit
for each module. We should to agree to use this format for all commit
messages and make commit detection easier.

I would drop the colon though to make it look like the usual 'git log
--oneline' output.

[1] http://stackoverflow.com/questions/223678/which-commit-has-this-blob

-- 

Kind regards, Yuriy.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2013-11-26 Thread Mark McLoughlin
On Fri, 2013-11-22 at 12:39 -0500, Doug Hellmann wrote:
 On Fri, Nov 22, 2013 at 4:11 AM, Flavio Percoco fla...@redhat.com wrote:
 
  Greetings,
 
  Based on the recent discussion that came out about not having enough
  information in the commit message when syncing oslo-incubator modules,
  I was thinking that besides encouraging people to write better commit
  messages, we could also improve the script we use to sync those
  modules.
 
  Some of the changes that I've been thinking of:
 
 1) Store the commit sha from which the module was copied from.
 Every project using oslo, currently keeps the list of modules it
 is using in `openstack-modules.conf` in a `module` parameter. We
 could store, along with the module name, the sha of the commit it
 was last synced from:
 
 module=log,commit
 
 or
 
 module=log
 log=commit
 
 
 The second form will be easier to manage. Humans edit the module field and
 the script will edit the others.

How about adding it as a comment at the end of the python files
themselves and leaving openstack-common.conf for human editing?

Mark.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2013-11-26 Thread Mark McLoughlin
On Fri, 2013-11-22 at 16:24 +, Duncan Thomas wrote:
 On 22 November 2013 14:59, Ben Nemec openst...@nemebean.com wrote:
 
  One other thought I had was to add the ability to split one Oslo sync up
  into multiple commits, either one per module, or even one per Oslo commit
  for some really large module changes (I'm thinking of the 1000 line db sync
  someone mentioned recently).  It would be more review churn, but at least it
  would keep the changes per review down to a more reasonable level.   I'm not
  positive it would be beneficial, but I thought I'd mention it.
 
 Cinder (often but not always me) tends to reject merges that do more
 that one module at a time, because it makes it far harder to review
 and spot problems, so some from of automation of this would be great.

Yes, it's good practice to sync related changes in individual commits,
rather than one big oslo sync.

An example from a previous sync I did in Cinder:

  https://review.openstack.org/#/c/12409/ (cfg)
  https://review.openstack.org/#/c/12411/ (zmq)
  https://review.openstack.org/#/c/12410/ (notifier)
  https://review.openstack.org/#/c/12412/ (misc)

These were all proposed at the same time, but I split related notable
changes into their own commits and then had a misc sync for more
trivial stuff.

Mark.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2013-11-25 Thread Flavio Percoco

On 25/11/13 09:34 +0800, Zhongyue Luo wrote:

Just a thought but couldn't the changes of a module update be calculated by
comparing the last commit dates of the source and target module?

For instance, if module A's update patch for Nova was uploaded on date XX then
we can filter out the changes from XX~present and print it out for the author
to paste in the commit message when running update.py for module A.

This way we might not need any changes to the openstack-modules.conf?


Not sure that would work. The update date of module A in project X
doesn't mean the project is at its latest version. I could've updated
an older version because the latest would've broken project X.

Cheers,
FF



On Sun, Nov 24, 2013 at 12:54 AM, Doug Hellmann doug.hellm...@dreamhost.com
wrote:

   Thanks for the reminder, Sandy.

   https://bugs.launchpad.net/oslo/+bug/1254300


   On Sat, Nov 23, 2013 at 9:39 AM, Sandy Walsh sandy.wa...@rackspace.com
   wrote:

   Seeing this thread reminded me:

   We need support in the update script for entry points in olso setup.cfg
   to make their way into the target project.

   So, if update is getting some love, please keep that in mind.
   ___
   OpenStack-dev mailing list
   OpenStack-dev@lists.openstack.org
   http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



   ___
   OpenStack-dev mailing list
   OpenStack-dev@lists.openstack.org
   http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev





--
Intel SSG/STO/DCST/CIT
880 Zixing Road, Zizhu Science Park, Minhang District, 200241, Shanghai, China
+862161166500



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
@flaper87
Flavio Percoco


pgpxpMTb75cQH.pgp
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2013-11-25 Thread Flavio Percoco

On 23/11/13 11:54 -0500, Doug Hellmann wrote:

Thanks for the reminder, Sandy.

https://bugs.launchpad.net/oslo/+bug/1254300


On Sat, Nov 23, 2013 at 9:39 AM, Sandy Walsh sandy.wa...@rackspace.com wrote:

   Seeing this thread reminded me:

   We need support in the update script for entry points in olso setup.cfg to
   make their way into the target project.

   So, if update is getting some love, please keep that in mind.



Good point, +1

I created this blueprint[0] and assigned it to me. I'll give some love
to update.py and get it done for I-2.

Cheers,
FF

[0] https://blueprints.launchpad.net/oslo/+spec/improve-update-script

--
@flaper87
Flavio Percoco


pgp610yEgnGwn.pgp
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2013-11-24 Thread Zhongyue Luo
Just a thought but couldn't the changes of a module update be calculated by
comparing the last commit dates of the source and target module?

For instance, if module A's update patch for Nova was uploaded on date XX
then we can filter out the changes from XX~present and print it out for the
author to paste in the commit message when running update.py for module A.

This way we might not need any changes to the openstack-modules.conf?


On Sun, Nov 24, 2013 at 12:54 AM, Doug Hellmann doug.hellm...@dreamhost.com
 wrote:

 Thanks for the reminder, Sandy.

 https://bugs.launchpad.net/oslo/+bug/1254300


 On Sat, Nov 23, 2013 at 9:39 AM, Sandy Walsh sandy.wa...@rackspace.comwrote:

 Seeing this thread reminded me:

 We need support in the update script for entry points in olso setup.cfg
 to make their way into the target project.

 So, if update is getting some love, please keep that in mind.
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 
*Intel SSG/STO/DCST/CIT*
880 Zixing Road, Zizhu Science Park, Minhang District, 200241, Shanghai,
China
+862161166500
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2013-11-23 Thread Sandy Walsh
Seeing this thread reminded me: 

We need support in the update script for entry points in olso setup.cfg to make 
their way into the target project.

So, if update is getting some love, please keep that in mind. 
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2013-11-23 Thread Doug Hellmann
Thanks for the reminder, Sandy.

https://bugs.launchpad.net/oslo/+bug/1254300


On Sat, Nov 23, 2013 at 9:39 AM, Sandy Walsh sandy.wa...@rackspace.comwrote:

 Seeing this thread reminded me:

 We need support in the update script for entry points in olso setup.cfg to
 make their way into the target project.

 So, if update is getting some love, please keep that in mind.
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Oslo] Improving oslo-incubator update.py

2013-11-22 Thread Flavio Percoco

Greetings,

Based on the recent discussion that came out about not having enough
information in the commit message when syncing oslo-incubator modules,
I was thinking that besides encouraging people to write better commit
messages, we could also improve the script we use to sync those
modules.

Some of the changes that I've been thinking of:

   1) Store the commit sha from which the module was copied from.
   Every project using oslo, currently keeps the list of modules it
   is using in `openstack-modules.conf` in a `module` parameter. We
   could store, along with the module name, the sha of the commit it
   was last synced from:

   module=log,commit

   or

   module=log
   log=commit

   2) Add an 'auto-commit' parameter to the update script that will
   generate a commit message with the short log of the commits where
   the modules being updated were modified. Soemthing like:

   Syncing oslo-incubator modules

   log.py:
   commit1: short-message
   commit2: short-message
   commit3: short-message

   lockutils:
   commit4: short-message
   commit5: short-message

#1 will help with figuring out when was the last time a module was
updated and what changes have been introduced since then.

#2 won't be the recommended way for writing commit messages. Oslo
incubator syncs can have different side-effects in every project.
However, it could be useful for trivial syncs.

Thoughts?

Cheers,
FF

--
@flaper87
Flavio Percoco


pgpCmV9OwmxXq.pgp
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2013-11-22 Thread Ben Nemec

On 2013-11-22 03:11, Flavio Percoco wrote:

Greetings,

Based on the recent discussion that came out about not having enough
information in the commit message when syncing oslo-incubator modules,
I was thinking that besides encouraging people to write better commit
messages, we could also improve the script we use to sync those
modules.

Some of the changes that I've been thinking of:

   1) Store the commit sha from which the module was copied from.
   Every project using oslo, currently keeps the list of modules it
   is using in `openstack-modules.conf` in a `module` parameter. We
   could store, along with the module name, the sha of the commit it
   was last synced from:

   module=log,commit

   or

   module=log
   log=commit

   2) Add an 'auto-commit' parameter to the update script that will
   generate a commit message with the short log of the commits where
   the modules being updated were modified. Soemthing like:

   Syncing oslo-incubator modules

   log.py:
   commit1: short-message
   commit2: short-message
   commit3: short-message

   lockutils:
   commit4: short-message
   commit5: short-message

#1 will help with figuring out when was the last time a module was
updated and what changes have been introduced since then.

#2 won't be the recommended way for writing commit messages. Oslo
incubator syncs can have different side-effects in every project.
However, it could be useful for trivial syncs.

Thoughts?


+1.  I've been thinking along the same lines.  I always try to include 
the git oneline output in my Oslo syncs anyway, but it's not trivial to 
figure out the appropriate ones to include so doing it automatically 
would be fantastic.


One other thought I had was to add the ability to split one Oslo sync up 
into multiple commits, either one per module, or even one per Oslo 
commit for some really large module changes (I'm thinking of the 1000 
line db sync someone mentioned recently).  It would be more review 
churn, but at least it would keep the changes per review down to a more 
reasonable level.   I'm not positive it would be beneficial, but I 
thought I'd mention it.


-Ben

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2013-11-22 Thread Duncan Thomas
On 22 November 2013 14:59, Ben Nemec openst...@nemebean.com wrote:

 One other thought I had was to add the ability to split one Oslo sync up
 into multiple commits, either one per module, or even one per Oslo commit
 for some really large module changes (I'm thinking of the 1000 line db sync
 someone mentioned recently).  It would be more review churn, but at least it
 would keep the changes per review down to a more reasonable level.   I'm not
 positive it would be beneficial, but I thought I'd mention it.

Cinder (often but not always me) tends to reject merges that do more
that one module at a time, because it makes it far harder to review
and spot problems, so some from of automation of this would be great.

-- 
Duncan Thomas

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2013-11-22 Thread Doug Hellmann
On Fri, Nov 22, 2013 at 4:11 AM, Flavio Percoco fla...@redhat.com wrote:

 Greetings,

 Based on the recent discussion that came out about not having enough
 information in the commit message when syncing oslo-incubator modules,
 I was thinking that besides encouraging people to write better commit
 messages, we could also improve the script we use to sync those
 modules.

 Some of the changes that I've been thinking of:

1) Store the commit sha from which the module was copied from.
Every project using oslo, currently keeps the list of modules it
is using in `openstack-modules.conf` in a `module` parameter. We
could store, along with the module name, the sha of the commit it
was last synced from:

module=log,commit

or

module=log
log=commit


The second form will be easier to manage. Humans edit the module field and
the script will edit the others.




2) Add an 'auto-commit' parameter to the update script that will
generate a commit message with the short log of the commits where
the modules being updated were modified. Soemthing like:

Syncing oslo-incubator modules

log.py:
commit1: short-message
commit2: short-message
commit3: short-message

lockutils:
commit4: short-message
commit5: short-message

 #1 will help with figuring out when was the last time a module was
 updated and what changes have been introduced since then.

 #2 won't be the recommended way for writing commit messages. Oslo
 incubator syncs can have different side-effects in every project.
 However, it could be useful for trivial syncs.


It looks like #2 would be a good way to start with commit messages, and
just leave it up to the person doing the merge to add the why information.

Doug




 Thoughts?

 Cheers,
 FF

 --
 @flaper87
 Flavio Percoco

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Oslo] Improving oslo-incubator update.py

2013-11-22 Thread Doug Hellmann
On Fri, Nov 22, 2013 at 11:24 AM, Duncan Thomas duncan.tho...@gmail.comwrote:

 On 22 November 2013 14:59, Ben Nemec openst...@nemebean.com wrote:

  One other thought I had was to add the ability to split one Oslo sync up
  into multiple commits, either one per module, or even one per Oslo commit
  for some really large module changes (I'm thinking of the 1000 line db
 sync
  someone mentioned recently).  It would be more review churn, but at
 least it
  would keep the changes per review down to a more reasonable level.   I'm
 not
  positive it would be beneficial, but I thought I'd mention it.

 Cinder (often but not always me) tends to reject merges that do more
 that one module at a time, because it makes it far harder to review
 and spot problems, so some from of automation of this would be great.


There are times when the commits are related, but in general this seems
like a good practice.

Doug




 --
 Duncan Thomas

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev