Re: [openstack-dev] [fuel] librarian-puppet integration, need help with build tasks for fuel-library
Do we have any success here.. ? On Mon, Jul 20, 2015 at 8:32 AM Alex Schultz aschu...@mirantis.com wrote: Vladimir, Thanks. Can you point me to the error for perestroika? I'd be happy to take a look as well. I spent most of Friday throwing various options at the CI system to try and figure out how to get the spec to work with the CI fuel-library package building so perhaps there's a different way to handle this in the spec. -Alex On Mon, Jul 20, 2015 at 10:02 AM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, As I've just found out this package available here [1] is not actually build with your patch (instead it is from previous successful build). Looks like perestroika can not build this package due to some environment related issues. I've poked Dmitry Burmistrov to check it out. However, your patch is OK, make system can build this package and ISO passes BVT tests. [1] http://172.18.160.74/osci/review/CR-202763/mos/7.0/fuel/base/centos6/Packages/fuel-library7.0-7.0.0-1.mos6888.git.bac86fe.noarch.rpm Vladimir Kozhukalov On Mon, Jul 20, 2015 at 4:04 PM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Yes, spec is much better place to introduce this. BTW, perestroika builds new package for every commit and patch set and publishes them via HTTP. Please check here http://172.18.160.74/osci/review/CR-202763/mos/7.0/fuel/base/centos6/Packages/fuel-library7.0-7.0.0-1.mos6888.git.bac86fe.noarch.rpm if the package contains all necessary upstream modules. Vladimir Kozhukalov On Sat, Jul 18, 2015 at 3:28 AM, Alex Schultz aschu...@mirantis.com wrote: Not until we start using it then any ci that tests with that module will validate the modules inclusion. You can check the output of the jobs as we are printing what modules are managed by librarian. -Alex On Jul 17, 2015 6:17 PM, Andrew Woodward xar...@gmail.com wrote: Fantastic, do we have some way to validate that the module was pulled in properly as part of fuel-library CI? On Fri, Jul 17, 2015 at 2:48 PM Alex Schultz aschu...@mirantis.com wrote: Hey All, I've figured it out without having to modify the fuel-main build code. I've updated the fuel-library spec with a build action that invokes the script to pull down external modules. Please take some time to review the two reviews out there for this change to see if there are any issues with the way it is implemented. https://review.openstack.org/#/c/202763/ https://review.openstack.org/#/c/202767/ This is a first step towards being able to pull in unmodified external puppet modules. Thanks, -Alex On Fri, Jul 17, 2015 at 4:23 PM, Andrew Woodward awoodw...@mirantis.com wrote: On Fri, Jul 17, 2015 at 12:24 PM Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Great that you did this. Now I think I can prepare fuel-main patch to invoke this script right before building fuel-library package. I'll add you to review it. Is it ok if I do this monday morning? Keep in minde we agreeded to a deadline to get this sorted and in shape to land by EOD monday or we will have to retain the old, and crappy fork method. If possible please work out how this needs to work as early as possible so Alex can continue. Vladimir Kozhukalov On Fri, Jul 17, 2015 at 5:51 PM, Alex Schultz aschu...@mirantis.com wrote: Hey Vladimir, On Fri, Jul 17, 2015 at 7:33 AM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Gathering upstream modules certainly should be implemented as a separate script so as to make it possible to use it wherever we need this (tests, builds, etc.) According to builds there are two things 1) We have so called perestroika package build system (Dmitry Burmistrov is a main contributor here). By the end of next week we are going to switch building all the packages to perestroika. And in order to gather upstream modules right before building fuel-library package, we need to change perestroika build scripts. 2) Currently we build packages using make system and you are right about the place where you need to make changes. https://github.com/stackforge/fuel-main/blob/master/packages/module.mk#L63-L82 If you create shell script, I'll help you to add it to make code. I have updated my review[0] to extract the update logic to it's own bash script that can be invoked by the build scripts. Let me know what would be the best way to wedge this in there. I think for the perestroika this would also be needed for the fuel-library build, so if you point me at that I can see if I can help make that change as well. Thanks, -Alex [0] https://review.openstack.org/#/c/202763/ Vladimir Kozhukalov On Fri, Jul 17, 2015 at 2:56 PM, Aleksandr Didenko adide...@mirantis.com wrote: I believe build_repo function is the best way to do this [0]. So for fuel-library we'll need to run a shell script right from the repo before 'touch $$@'. We can make
Re: [openstack-dev] [fuel] librarian-puppet integration, need help with build tasks for fuel-library
Just to followup since the required packages are finally available, the patches have been updated and are passing CI now. https://review.openstack.org/#/c/202763/ -Alex On Fri, Jul 24, 2015 at 8:32 AM, Alex Schultz aschu...@mirantis.com wrote: Unfortunately we got stuck with package availability issues. It was successful without using librarian packages for building, but since we need to leverage packages the work to get packaged versions of the code has unfortunately blocked this. I've got the packages built, but the time to get them to an publicly available mirror has killed this effort. Since it's now past FF, I don't think this will be for 7.0 but we should be in a good spot to start early for 8.0. Also I've switched to librarian-puppet-simple since it has far fewer dependencies and we only want to support git references for packages. Once that makes it to the mirror and can pass CI, this change would in theory be ready. I'm continuing to keep an eye on the changes so that we'll be ready to start running with them when the time comes. -Alex On Fri, Jul 24, 2015 at 1:41 AM, Mike Scherbakov mscherba...@mirantis.com wrote: Do we have any success here.. ? On Mon, Jul 20, 2015 at 8:32 AM Alex Schultz aschu...@mirantis.com wrote: Vladimir, Thanks. Can you point me to the error for perestroika? I'd be happy to take a look as well. I spent most of Friday throwing various options at the CI system to try and figure out how to get the spec to work with the CI fuel-library package building so perhaps there's a different way to handle this in the spec. -Alex On Mon, Jul 20, 2015 at 10:02 AM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, As I've just found out this package available here [1] is not actually build with your patch (instead it is from previous successful build). Looks like perestroika can not build this package due to some environment related issues. I've poked Dmitry Burmistrov to check it out. However, your patch is OK, make system can build this package and ISO passes BVT tests. [1] http://172.18.160.74/osci/review/CR-202763/mos/7.0/fuel/base/centos6/Packages/fuel-library7.0-7.0.0-1.mos6888.git.bac86fe.noarch.rpm Vladimir Kozhukalov On Mon, Jul 20, 2015 at 4:04 PM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Yes, spec is much better place to introduce this. BTW, perestroika builds new package for every commit and patch set and publishes them via HTTP. Please check here http://172.18.160.74/osci/review/CR-202763/mos/7.0/fuel/base/centos6/Packages/fuel-library7.0-7.0.0-1.mos6888.git.bac86fe.noarch.rpm if the package contains all necessary upstream modules. Vladimir Kozhukalov On Sat, Jul 18, 2015 at 3:28 AM, Alex Schultz aschu...@mirantis.com wrote: Not until we start using it then any ci that tests with that module will validate the modules inclusion. You can check the output of the jobs as we are printing what modules are managed by librarian. -Alex On Jul 17, 2015 6:17 PM, Andrew Woodward xar...@gmail.com wrote: Fantastic, do we have some way to validate that the module was pulled in properly as part of fuel-library CI? On Fri, Jul 17, 2015 at 2:48 PM Alex Schultz aschu...@mirantis.com wrote: Hey All, I've figured it out without having to modify the fuel-main build code. I've updated the fuel-library spec with a build action that invokes the script to pull down external modules. Please take some time to review the two reviews out there for this change to see if there are any issues with the way it is implemented. https://review.openstack.org/#/c/202763/ https://review.openstack.org/#/c/202767/ This is a first step towards being able to pull in unmodified external puppet modules. Thanks, -Alex On Fri, Jul 17, 2015 at 4:23 PM, Andrew Woodward awoodw...@mirantis.com wrote: On Fri, Jul 17, 2015 at 12:24 PM Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Great that you did this. Now I think I can prepare fuel-main patch to invoke this script right before building fuel-library package. I'll add you to review it. Is it ok if I do this monday morning? Keep in minde we agreeded to a deadline to get this sorted and in shape to land by EOD monday or we will have to retain the old, and crappy fork method. If possible please work out how this needs to work as early as possible so Alex can continue. Vladimir Kozhukalov On Fri, Jul 17, 2015 at 5:51 PM, Alex Schultz aschu...@mirantis.com wrote: Hey Vladimir, On Fri, Jul 17, 2015 at 7:33 AM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Gathering upstream modules certainly should be implemented as a separate script so as to make it possible to use it wherever we need this (tests, builds, etc.) According to builds there are two things 1) We have so called perestroika package build system (Dmitry Burmistrov is a main contributor here). By the end of next
Re: [openstack-dev] [fuel] librarian-puppet integration, need help with build tasks for fuel-library
Unfortunately we got stuck with package availability issues. It was successful without using librarian packages for building, but since we need to leverage packages the work to get packaged versions of the code has unfortunately blocked this. I've got the packages built, but the time to get them to an publicly available mirror has killed this effort. Since it's now past FF, I don't think this will be for 7.0 but we should be in a good spot to start early for 8.0. Also I've switched to librarian-puppet-simple since it has far fewer dependencies and we only want to support git references for packages. Once that makes it to the mirror and can pass CI, this change would in theory be ready. I'm continuing to keep an eye on the changes so that we'll be ready to start running with them when the time comes. -Alex On Fri, Jul 24, 2015 at 1:41 AM, Mike Scherbakov mscherba...@mirantis.com wrote: Do we have any success here.. ? On Mon, Jul 20, 2015 at 8:32 AM Alex Schultz aschu...@mirantis.com wrote: Vladimir, Thanks. Can you point me to the error for perestroika? I'd be happy to take a look as well. I spent most of Friday throwing various options at the CI system to try and figure out how to get the spec to work with the CI fuel-library package building so perhaps there's a different way to handle this in the spec. -Alex On Mon, Jul 20, 2015 at 10:02 AM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, As I've just found out this package available here [1] is not actually build with your patch (instead it is from previous successful build). Looks like perestroika can not build this package due to some environment related issues. I've poked Dmitry Burmistrov to check it out. However, your patch is OK, make system can build this package and ISO passes BVT tests. [1] http://172.18.160.74/osci/review/CR-202763/mos/7.0/fuel/base/centos6/Packages/fuel-library7.0-7.0.0-1.mos6888.git.bac86fe.noarch.rpm Vladimir Kozhukalov On Mon, Jul 20, 2015 at 4:04 PM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Yes, spec is much better place to introduce this. BTW, perestroika builds new package for every commit and patch set and publishes them via HTTP. Please check here http://172.18.160.74/osci/review/CR-202763/mos/7.0/fuel/base/centos6/Packages/fuel-library7.0-7.0.0-1.mos6888.git.bac86fe.noarch.rpm if the package contains all necessary upstream modules. Vladimir Kozhukalov On Sat, Jul 18, 2015 at 3:28 AM, Alex Schultz aschu...@mirantis.com wrote: Not until we start using it then any ci that tests with that module will validate the modules inclusion. You can check the output of the jobs as we are printing what modules are managed by librarian. -Alex On Jul 17, 2015 6:17 PM, Andrew Woodward xar...@gmail.com wrote: Fantastic, do we have some way to validate that the module was pulled in properly as part of fuel-library CI? On Fri, Jul 17, 2015 at 2:48 PM Alex Schultz aschu...@mirantis.com wrote: Hey All, I've figured it out without having to modify the fuel-main build code. I've updated the fuel-library spec with a build action that invokes the script to pull down external modules. Please take some time to review the two reviews out there for this change to see if there are any issues with the way it is implemented. https://review.openstack.org/#/c/202763/ https://review.openstack.org/#/c/202767/ This is a first step towards being able to pull in unmodified external puppet modules. Thanks, -Alex On Fri, Jul 17, 2015 at 4:23 PM, Andrew Woodward awoodw...@mirantis.com wrote: On Fri, Jul 17, 2015 at 12:24 PM Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Great that you did this. Now I think I can prepare fuel-main patch to invoke this script right before building fuel-library package. I'll add you to review it. Is it ok if I do this monday morning? Keep in minde we agreeded to a deadline to get this sorted and in shape to land by EOD monday or we will have to retain the old, and crappy fork method. If possible please work out how this needs to work as early as possible so Alex can continue. Vladimir Kozhukalov On Fri, Jul 17, 2015 at 5:51 PM, Alex Schultz aschu...@mirantis.com wrote: Hey Vladimir, On Fri, Jul 17, 2015 at 7:33 AM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Gathering upstream modules certainly should be implemented as a separate script so as to make it possible to use it wherever we need this (tests, builds, etc.) According to builds there are two things 1) We have so called perestroika package build system (Dmitry Burmistrov is a main contributor here). By the end of next week we are going to switch building all the packages to perestroika. And in order to gather upstream modules right before building fuel-library package, we need to change perestroika build scripts. 2) Currently we build packages using make system and you
Re: [openstack-dev] [fuel] librarian-puppet integration, need help with build tasks for fuel-library
Alex, Yes, spec is much better place to introduce this. BTW, perestroika builds new package for every commit and patch set and publishes them via HTTP. Please check here http://172.18.160.74/osci/review/CR-202763/mos/7.0/fuel/base/centos6/Packages/fuel-library7.0-7.0.0-1.mos6888.git.bac86fe.noarch.rpm if the package contains all necessary upstream modules. Vladimir Kozhukalov On Sat, Jul 18, 2015 at 3:28 AM, Alex Schultz aschu...@mirantis.com wrote: Not until we start using it then any ci that tests with that module will validate the modules inclusion. You can check the output of the jobs as we are printing what modules are managed by librarian. -Alex On Jul 17, 2015 6:17 PM, Andrew Woodward xar...@gmail.com wrote: Fantastic, do we have some way to validate that the module was pulled in properly as part of fuel-library CI? On Fri, Jul 17, 2015 at 2:48 PM Alex Schultz aschu...@mirantis.com wrote: Hey All, I've figured it out without having to modify the fuel-main build code. I've updated the fuel-library spec with a build action that invokes the script to pull down external modules. Please take some time to review the two reviews out there for this change to see if there are any issues with the way it is implemented. https://review.openstack.org/#/c/202763/ https://review.openstack.org/#/c/202767/ This is a first step towards being able to pull in unmodified external puppet modules. Thanks, -Alex On Fri, Jul 17, 2015 at 4:23 PM, Andrew Woodward awoodw...@mirantis.com wrote: On Fri, Jul 17, 2015 at 12:24 PM Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Great that you did this. Now I think I can prepare fuel-main patch to invoke this script right before building fuel-library package. I'll add you to review it. Is it ok if I do this monday morning? Keep in minde we agreeded to a deadline to get this sorted and in shape to land by EOD monday or we will have to retain the old, and crappy fork method. If possible please work out how this needs to work as early as possible so Alex can continue. Vladimir Kozhukalov On Fri, Jul 17, 2015 at 5:51 PM, Alex Schultz aschu...@mirantis.com wrote: Hey Vladimir, On Fri, Jul 17, 2015 at 7:33 AM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Gathering upstream modules certainly should be implemented as a separate script so as to make it possible to use it wherever we need this (tests, builds, etc.) According to builds there are two things 1) We have so called perestroika package build system (Dmitry Burmistrov is a main contributor here). By the end of next week we are going to switch building all the packages to perestroika. And in order to gather upstream modules right before building fuel-library package, we need to change perestroika build scripts. 2) Currently we build packages using make system and you are right about the place where you need to make changes. https://github.com/stackforge/fuel-main/blob/master/packages/module.mk#L63-L82 If you create shell script, I'll help you to add it to make code. I have updated my review[0] to extract the update logic to it's own bash script that can be invoked by the build scripts. Let me know what would be the best way to wedge this in there. I think for the perestroika this would also be needed for the fuel-library build, so if you point me at that I can see if I can help make that change as well. Thanks, -Alex [0] https://review.openstack.org/#/c/202763/ Vladimir Kozhukalov On Fri, Jul 17, 2015 at 2:56 PM, Aleksandr Didenko adide...@mirantis.com wrote: I believe build_repo function is the best way to do this [0]. So for fuel-library we'll need to run a shell script right from the repo before 'touch $$@'. We can make it either conditional ( test -f ./path/additional_build_script.sh bash ./path/additional_build_script.sh ) or as additional parameter to function and add it in fuel-library call [1] Regards, Alex [0] https://github.com/stackforge/fuel-main/blob/master/repos.mk#L16-L37 [1] https://github.com/stackforge/fuel-main/blob/master/repos.mk#L45 On Fri, Jul 17, 2015 at 2:37 PM, Alex Schultz aschu...@mirantis.com wrote: Hey Alex, On Jul 17, 2015 4:32 AM, Aleksandr Didenko adide...@mirantis.com wrote: Hi, I think that we should provide a separate script that will fetch the upstream modules into fuel-library/deployment/puppet/ directory. It will allow us to have everything in a single place and use this script in ISO build process and CI jobs. Right. That is what I'm going for. The issue I need help with is the best way to execute this as part of the build process. From what i understand of the build process is that we are using git archive for all pieces so I'm not sure how to wedge in an extra script execution to do the module fetch. The creation of the script isn't the issue, the issue is how can I properly run it as part of
Re: [openstack-dev] [fuel] librarian-puppet integration, need help with build tasks for fuel-library
Alex, As I've just found out this package available here [1] is not actually build with your patch (instead it is from previous successful build). Looks like perestroika can not build this package due to some environment related issues. I've poked Dmitry Burmistrov to check it out. However, your patch is OK, make system can build this package and ISO passes BVT tests. [1] http://172.18.160.74/osci/review/CR-202763/mos/7.0/fuel/base/centos6/Packages/fuel-library7.0-7.0.0-1.mos6888.git.bac86fe.noarch.rpm Vladimir Kozhukalov On Mon, Jul 20, 2015 at 4:04 PM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Yes, spec is much better place to introduce this. BTW, perestroika builds new package for every commit and patch set and publishes them via HTTP. Please check here http://172.18.160.74/osci/review/CR-202763/mos/7.0/fuel/base/centos6/Packages/fuel-library7.0-7.0.0-1.mos6888.git.bac86fe.noarch.rpm if the package contains all necessary upstream modules. Vladimir Kozhukalov On Sat, Jul 18, 2015 at 3:28 AM, Alex Schultz aschu...@mirantis.com wrote: Not until we start using it then any ci that tests with that module will validate the modules inclusion. You can check the output of the jobs as we are printing what modules are managed by librarian. -Alex On Jul 17, 2015 6:17 PM, Andrew Woodward xar...@gmail.com wrote: Fantastic, do we have some way to validate that the module was pulled in properly as part of fuel-library CI? On Fri, Jul 17, 2015 at 2:48 PM Alex Schultz aschu...@mirantis.com wrote: Hey All, I've figured it out without having to modify the fuel-main build code. I've updated the fuel-library spec with a build action that invokes the script to pull down external modules. Please take some time to review the two reviews out there for this change to see if there are any issues with the way it is implemented. https://review.openstack.org/#/c/202763/ https://review.openstack.org/#/c/202767/ This is a first step towards being able to pull in unmodified external puppet modules. Thanks, -Alex On Fri, Jul 17, 2015 at 4:23 PM, Andrew Woodward awoodw...@mirantis.com wrote: On Fri, Jul 17, 2015 at 12:24 PM Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Great that you did this. Now I think I can prepare fuel-main patch to invoke this script right before building fuel-library package. I'll add you to review it. Is it ok if I do this monday morning? Keep in minde we agreeded to a deadline to get this sorted and in shape to land by EOD monday or we will have to retain the old, and crappy fork method. If possible please work out how this needs to work as early as possible so Alex can continue. Vladimir Kozhukalov On Fri, Jul 17, 2015 at 5:51 PM, Alex Schultz aschu...@mirantis.com wrote: Hey Vladimir, On Fri, Jul 17, 2015 at 7:33 AM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Gathering upstream modules certainly should be implemented as a separate script so as to make it possible to use it wherever we need this (tests, builds, etc.) According to builds there are two things 1) We have so called perestroika package build system (Dmitry Burmistrov is a main contributor here). By the end of next week we are going to switch building all the packages to perestroika. And in order to gather upstream modules right before building fuel-library package, we need to change perestroika build scripts. 2) Currently we build packages using make system and you are right about the place where you need to make changes. https://github.com/stackforge/fuel-main/blob/master/packages/module.mk#L63-L82 If you create shell script, I'll help you to add it to make code. I have updated my review[0] to extract the update logic to it's own bash script that can be invoked by the build scripts. Let me know what would be the best way to wedge this in there. I think for the perestroika this would also be needed for the fuel-library build, so if you point me at that I can see if I can help make that change as well. Thanks, -Alex [0] https://review.openstack.org/#/c/202763/ Vladimir Kozhukalov On Fri, Jul 17, 2015 at 2:56 PM, Aleksandr Didenko adide...@mirantis.com wrote: I believe build_repo function is the best way to do this [0]. So for fuel-library we'll need to run a shell script right from the repo before 'touch $$@'. We can make it either conditional ( test -f ./path/additional_build_script.sh bash ./path/additional_build_script.sh ) or as additional parameter to function and add it in fuel-library call [1] Regards, Alex [0] https://github.com/stackforge/fuel-main/blob/master/repos.mk#L16-L37 [1] https://github.com/stackforge/fuel-main/blob/master/repos.mk#L45 On Fri, Jul 17, 2015 at 2:37 PM, Alex Schultz aschu...@mirantis.com wrote: Hey Alex, On Jul 17, 2015 4:32 AM, Aleksandr Didenko adide...@mirantis.com wrote: Hi, I think that we should
Re: [openstack-dev] [fuel] librarian-puppet integration, need help with build tasks for fuel-library
Vladimir, Thanks. Can you point me to the error for perestroika? I'd be happy to take a look as well. I spent most of Friday throwing various options at the CI system to try and figure out how to get the spec to work with the CI fuel-library package building so perhaps there's a different way to handle this in the spec. -Alex On Mon, Jul 20, 2015 at 10:02 AM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, As I've just found out this package available here [1] is not actually build with your patch (instead it is from previous successful build). Looks like perestroika can not build this package due to some environment related issues. I've poked Dmitry Burmistrov to check it out. However, your patch is OK, make system can build this package and ISO passes BVT tests. [1] http://172.18.160.74/osci/review/CR-202763/mos/7.0/fuel/base/centos6/Packages/fuel-library7.0-7.0.0-1.mos6888.git.bac86fe.noarch.rpm Vladimir Kozhukalov On Mon, Jul 20, 2015 at 4:04 PM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Yes, spec is much better place to introduce this. BTW, perestroika builds new package for every commit and patch set and publishes them via HTTP. Please check here http://172.18.160.74/osci/review/CR-202763/mos/7.0/fuel/base/centos6/Packages/fuel-library7.0-7.0.0-1.mos6888.git.bac86fe.noarch.rpm if the package contains all necessary upstream modules. Vladimir Kozhukalov On Sat, Jul 18, 2015 at 3:28 AM, Alex Schultz aschu...@mirantis.com wrote: Not until we start using it then any ci that tests with that module will validate the modules inclusion. You can check the output of the jobs as we are printing what modules are managed by librarian. -Alex On Jul 17, 2015 6:17 PM, Andrew Woodward xar...@gmail.com wrote: Fantastic, do we have some way to validate that the module was pulled in properly as part of fuel-library CI? On Fri, Jul 17, 2015 at 2:48 PM Alex Schultz aschu...@mirantis.com wrote: Hey All, I've figured it out without having to modify the fuel-main build code. I've updated the fuel-library spec with a build action that invokes the script to pull down external modules. Please take some time to review the two reviews out there for this change to see if there are any issues with the way it is implemented. https://review.openstack.org/#/c/202763/ https://review.openstack.org/#/c/202767/ This is a first step towards being able to pull in unmodified external puppet modules. Thanks, -Alex On Fri, Jul 17, 2015 at 4:23 PM, Andrew Woodward awoodw...@mirantis.com wrote: On Fri, Jul 17, 2015 at 12:24 PM Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Great that you did this. Now I think I can prepare fuel-main patch to invoke this script right before building fuel-library package. I'll add you to review it. Is it ok if I do this monday morning? Keep in minde we agreeded to a deadline to get this sorted and in shape to land by EOD monday or we will have to retain the old, and crappy fork method. If possible please work out how this needs to work as early as possible so Alex can continue. Vladimir Kozhukalov On Fri, Jul 17, 2015 at 5:51 PM, Alex Schultz aschu...@mirantis.com wrote: Hey Vladimir, On Fri, Jul 17, 2015 at 7:33 AM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Gathering upstream modules certainly should be implemented as a separate script so as to make it possible to use it wherever we need this (tests, builds, etc.) According to builds there are two things 1) We have so called perestroika package build system (Dmitry Burmistrov is a main contributor here). By the end of next week we are going to switch building all the packages to perestroika. And in order to gather upstream modules right before building fuel-library package, we need to change perestroika build scripts. 2) Currently we build packages using make system and you are right about the place where you need to make changes. https://github.com/stackforge/fuel-main/blob/master/packages/module.mk#L63-L82 If you create shell script, I'll help you to add it to make code. I have updated my review[0] to extract the update logic to it's own bash script that can be invoked by the build scripts. Let me know what would be the best way to wedge this in there. I think for the perestroika this would also be needed for the fuel-library build, so if you point me at that I can see if I can help make that change as well. Thanks, -Alex [0] https://review.openstack.org/#/c/202763/ Vladimir Kozhukalov On Fri, Jul 17, 2015 at 2:56 PM, Aleksandr Didenko adide...@mirantis.com wrote: I believe build_repo function is the best way to do this [0]. So for fuel-library we'll need to run a shell script right from the repo before 'touch $$@'. We can make it either conditional ( test -f ./path/additional_build_script.sh bash ./path/additional_build_script.sh ) or as
Re: [openstack-dev] [fuel] librarian-puppet integration, need help with build tasks for fuel-library
Hey Alex, On Jul 17, 2015 4:32 AM, Aleksandr Didenko adide...@mirantis.com wrote: Hi, I think that we should provide a separate script that will fetch the upstream modules into fuel-library/deployment/puppet/ directory. It will allow us to have everything in a single place and use this script in ISO build process and CI jobs. Right. That is what I'm going for. The issue I need help with is the best way to execute this as part of the build process. From what i understand of the build process is that we are using git archive for all pieces so I'm not sure how to wedge in an extra script execution to do the module fetch. The creation of the script isn't the issue, the issue is how can I properly run it as part of the build process. Regards, Alex Thanks, -Alex On Thu, Jul 16, 2015 at 11:17 PM, Alex Schultz aschu...@mirantis.com wrote: Hello everyone, I have committed the initial configuration required to start leveraging librarian-puppet as part of the way we pull in upstream puppet modules[0]. Additionally, I have also committed a change that would pull in the openstack-ironic module[1]. The one piece that is missing from this being a complete solution is the ability to run librarian-puppet as part of our build process for the fuel-library. I've looked into the fuel-main build scripts and I think it's over my head to figure this out just by looking. Can anyone explain to me or assist me in how I could go about modifying the existing build system to be able to run librarian-puppet to prepare the source for the package? In my initial investigation, it looks like it would be a modification of the fuel-main/packages/module.mk[3] file. I basically need to do the prepare_library[3] function from the 202763 review[0] after we've pulled all the sources together to fetch the upstream modules. Thanks, -Alex [0] https://review.openstack.org/202763 [1] https://review.openstack.org/202767 [2] https://github.com/stackforge/fuel-main/blob/master/packages/module.mk#L63-L82 [3] https://review.openstack.org/#/c/202763/1/utils/jenkins/fuel_noop_tests.rb __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [fuel] librarian-puppet integration, need help with build tasks for fuel-library
I believe build_repo function is the best way to do this [0]. So for fuel-library we'll need to run a shell script right from the repo before 'touch $$@'. We can make it either conditional ( test -f ./path/additional_build_script.sh bash ./path/additional_build_script.sh ) or as additional parameter to function and add it in fuel-library call [1] Regards, Alex [0] https://github.com/stackforge/fuel-main/blob/master/repos.mk#L16-L37 [1] https://github.com/stackforge/fuel-main/blob/master/repos.mk#L45 On Fri, Jul 17, 2015 at 2:37 PM, Alex Schultz aschu...@mirantis.com wrote: Hey Alex, On Jul 17, 2015 4:32 AM, Aleksandr Didenko adide...@mirantis.com wrote: Hi, I think that we should provide a separate script that will fetch the upstream modules into fuel-library/deployment/puppet/ directory. It will allow us to have everything in a single place and use this script in ISO build process and CI jobs. Right. That is what I'm going for. The issue I need help with is the best way to execute this as part of the build process. From what i understand of the build process is that we are using git archive for all pieces so I'm not sure how to wedge in an extra script execution to do the module fetch. The creation of the script isn't the issue, the issue is how can I properly run it as part of the build process. Regards, Alex Thanks, -Alex On Thu, Jul 16, 2015 at 11:17 PM, Alex Schultz aschu...@mirantis.com wrote: Hello everyone, I have committed the initial configuration required to start leveraging librarian-puppet as part of the way we pull in upstream puppet modules[0]. Additionally, I have also committed a change that would pull in the openstack-ironic module[1]. The one piece that is missing from this being a complete solution is the ability to run librarian-puppet as part of our build process for the fuel-library. I've looked into the fuel-main build scripts and I think it's over my head to figure this out just by looking. Can anyone explain to me or assist me in how I could go about modifying the existing build system to be able to run librarian-puppet to prepare the source for the package? In my initial investigation, it looks like it would be a modification of the fuel-main/packages/module.mk[3] file. I basically need to do the prepare_library[3] function from the 202763 review[0] after we've pulled all the sources together to fetch the upstream modules. Thanks, -Alex [0] https://review.openstack.org/202763 [1] https://review.openstack.org/202767 [2] https://github.com/stackforge/fuel-main/blob/master/packages/module.mk#L63-L82 [3] https://review.openstack.org/#/c/202763/1/utils/jenkins/fuel_noop_tests.rb __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [fuel] librarian-puppet integration, need help with build tasks for fuel-library
Alex, Gathering upstream modules certainly should be implemented as a separate script so as to make it possible to use it wherever we need this (tests, builds, etc.) According to builds there are two things 1) We have so called perestroika package build system (Dmitry Burmistrov is a main contributor here). By the end of next week we are going to switch building all the packages to perestroika. And in order to gather upstream modules right before building fuel-library package, we need to change perestroika build scripts. 2) Currently we build packages using make system and you are right about the place where you need to make changes. https://github.com/stackforge/fuel-main/blob/master/packages/module.mk#L63-L82 If you create shell script, I'll help you to add it to make code. Vladimir Kozhukalov On Fri, Jul 17, 2015 at 2:56 PM, Aleksandr Didenko adide...@mirantis.com wrote: I believe build_repo function is the best way to do this [0]. So for fuel-library we'll need to run a shell script right from the repo before 'touch $$@'. We can make it either conditional ( test -f ./path/additional_build_script.sh bash ./path/additional_build_script.sh ) or as additional parameter to function and add it in fuel-library call [1] Regards, Alex [0] https://github.com/stackforge/fuel-main/blob/master/repos.mk#L16-L37 [1] https://github.com/stackforge/fuel-main/blob/master/repos.mk#L45 On Fri, Jul 17, 2015 at 2:37 PM, Alex Schultz aschu...@mirantis.com wrote: Hey Alex, On Jul 17, 2015 4:32 AM, Aleksandr Didenko adide...@mirantis.com wrote: Hi, I think that we should provide a separate script that will fetch the upstream modules into fuel-library/deployment/puppet/ directory. It will allow us to have everything in a single place and use this script in ISO build process and CI jobs. Right. That is what I'm going for. The issue I need help with is the best way to execute this as part of the build process. From what i understand of the build process is that we are using git archive for all pieces so I'm not sure how to wedge in an extra script execution to do the module fetch. The creation of the script isn't the issue, the issue is how can I properly run it as part of the build process. Regards, Alex Thanks, -Alex On Thu, Jul 16, 2015 at 11:17 PM, Alex Schultz aschu...@mirantis.com wrote: Hello everyone, I have committed the initial configuration required to start leveraging librarian-puppet as part of the way we pull in upstream puppet modules[0]. Additionally, I have also committed a change that would pull in the openstack-ironic module[1]. The one piece that is missing from this being a complete solution is the ability to run librarian-puppet as part of our build process for the fuel-library. I've looked into the fuel-main build scripts and I think it's over my head to figure this out just by looking. Can anyone explain to me or assist me in how I could go about modifying the existing build system to be able to run librarian-puppet to prepare the source for the package? In my initial investigation, it looks like it would be a modification of the fuel-main/packages/module.mk[3] file. I basically need to do the prepare_library[3] function from the 202763 review[0] after we've pulled all the sources together to fetch the upstream modules. Thanks, -Alex [0] https://review.openstack.org/202763 [1] https://review.openstack.org/202767 [2] https://github.com/stackforge/fuel-main/blob/master/packages/module.mk#L63-L82 [3] https://review.openstack.org/#/c/202763/1/utils/jenkins/fuel_noop_tests.rb __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe:
Re: [openstack-dev] [fuel] librarian-puppet integration, need help with build tasks for fuel-library
Hey Vladimir, On Fri, Jul 17, 2015 at 7:33 AM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Gathering upstream modules certainly should be implemented as a separate script so as to make it possible to use it wherever we need this (tests, builds, etc.) According to builds there are two things 1) We have so called perestroika package build system (Dmitry Burmistrov is a main contributor here). By the end of next week we are going to switch building all the packages to perestroika. And in order to gather upstream modules right before building fuel-library package, we need to change perestroika build scripts. 2) Currently we build packages using make system and you are right about the place where you need to make changes. https://github.com/stackforge/fuel-main/blob/master/packages/module.mk#L63-L82 If you create shell script, I'll help you to add it to make code. I have updated my review[0] to extract the update logic to it's own bash script that can be invoked by the build scripts. Let me know what would be the best way to wedge this in there. I think for the perestroika this would also be needed for the fuel-library build, so if you point me at that I can see if I can help make that change as well. Thanks, -Alex [0] https://review.openstack.org/#/c/202763/ Vladimir Kozhukalov On Fri, Jul 17, 2015 at 2:56 PM, Aleksandr Didenko adide...@mirantis.com wrote: I believe build_repo function is the best way to do this [0]. So for fuel-library we'll need to run a shell script right from the repo before 'touch $$@'. We can make it either conditional ( test -f ./path/additional_build_script.sh bash ./path/additional_build_script.sh ) or as additional parameter to function and add it in fuel-library call [1] Regards, Alex [0] https://github.com/stackforge/fuel-main/blob/master/repos.mk#L16-L37 [1] https://github.com/stackforge/fuel-main/blob/master/repos.mk#L45 On Fri, Jul 17, 2015 at 2:37 PM, Alex Schultz aschu...@mirantis.com wrote: Hey Alex, On Jul 17, 2015 4:32 AM, Aleksandr Didenko adide...@mirantis.com wrote: Hi, I think that we should provide a separate script that will fetch the upstream modules into fuel-library/deployment/puppet/ directory. It will allow us to have everything in a single place and use this script in ISO build process and CI jobs. Right. That is what I'm going for. The issue I need help with is the best way to execute this as part of the build process. From what i understand of the build process is that we are using git archive for all pieces so I'm not sure how to wedge in an extra script execution to do the module fetch. The creation of the script isn't the issue, the issue is how can I properly run it as part of the build process. Regards, Alex Thanks, -Alex On Thu, Jul 16, 2015 at 11:17 PM, Alex Schultz aschu...@mirantis.com wrote: Hello everyone, I have committed the initial configuration required to start leveraging librarian-puppet as part of the way we pull in upstream puppet modules[0]. Additionally, I have also committed a change that would pull in the openstack-ironic module[1]. The one piece that is missing from this being a complete solution is the ability to run librarian-puppet as part of our build process for the fuel-library. I've looked into the fuel-main build scripts and I think it's over my head to figure this out just by looking. Can anyone explain to me or assist me in how I could go about modifying the existing build system to be able to run librarian-puppet to prepare the source for the package? In my initial investigation, it looks like it would be a modification of the fuel-main/packages/module.mk[3] file. I basically need to do the prepare_library[3] function from the 202763 review[0] after we've pulled all the sources together to fetch the upstream modules. Thanks, -Alex [0] https://review.openstack.org/202763 [1] https://review.openstack.org/202767 [2] https://github.com/stackforge/fuel-main/blob/master/packages/module.mk#L63-L82 [3] https://review.openstack.org/#/c/202763/1/utils/jenkins/fuel_noop_tests.rb __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
Re: [openstack-dev] [fuel] librarian-puppet integration, need help with build tasks for fuel-library
Hi, I think that we should provide a separate script that will fetch the upstream modules into fuel-library/deployment/puppet/ directory. It will allow us to have everything in a single place and use this script in ISO build process and CI jobs. Regards, Alex On Thu, Jul 16, 2015 at 11:17 PM, Alex Schultz aschu...@mirantis.com wrote: Hello everyone, I have committed the initial configuration required to start leveraging librarian-puppet as part of the way we pull in upstream puppet modules[0]. Additionally, I have also committed a change that would pull in the openstack-ironic module[1]. The one piece that is missing from this being a complete solution is the ability to run librarian-puppet as part of our build process for the fuel-library. I've looked into the fuel-main build scripts and I think it's over my head to figure this out just by looking. Can anyone explain to me or assist me in how I could go about modifying the existing build system to be able to run librarian-puppet to prepare the source for the package? In my initial investigation, it looks like it would be a modification of the fuel-main/packages/module.mk[3] file. I basically need to do the prepare_library[3] function from the 202763 review[0] after we've pulled all the sources together to fetch the upstream modules. Thanks, -Alex [0] https://review.openstack.org/202763 [1] https://review.openstack.org/202767 [2] https://github.com/stackforge/fuel-main/blob/master/packages/module.mk#L63-L82 [3] https://review.openstack.org/#/c/202763/1/utils/jenkins/fuel_noop_tests.rb __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [fuel] librarian-puppet integration, need help with build tasks for fuel-library
Alex, Great that you did this. Now I think I can prepare fuel-main patch to invoke this script right before building fuel-library package. I'll add you to review it. Is it ok if I do this monday morning? Vladimir Kozhukalov On Fri, Jul 17, 2015 at 5:51 PM, Alex Schultz aschu...@mirantis.com wrote: Hey Vladimir, On Fri, Jul 17, 2015 at 7:33 AM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Gathering upstream modules certainly should be implemented as a separate script so as to make it possible to use it wherever we need this (tests, builds, etc.) According to builds there are two things 1) We have so called perestroika package build system (Dmitry Burmistrov is a main contributor here). By the end of next week we are going to switch building all the packages to perestroika. And in order to gather upstream modules right before building fuel-library package, we need to change perestroika build scripts. 2) Currently we build packages using make system and you are right about the place where you need to make changes. https://github.com/stackforge/fuel-main/blob/master/packages/module.mk#L63-L82 If you create shell script, I'll help you to add it to make code. I have updated my review[0] to extract the update logic to it's own bash script that can be invoked by the build scripts. Let me know what would be the best way to wedge this in there. I think for the perestroika this would also be needed for the fuel-library build, so if you point me at that I can see if I can help make that change as well. Thanks, -Alex [0] https://review.openstack.org/#/c/202763/ Vladimir Kozhukalov On Fri, Jul 17, 2015 at 2:56 PM, Aleksandr Didenko adide...@mirantis.com wrote: I believe build_repo function is the best way to do this [0]. So for fuel-library we'll need to run a shell script right from the repo before 'touch $$@'. We can make it either conditional ( test -f ./path/additional_build_script.sh bash ./path/additional_build_script.sh ) or as additional parameter to function and add it in fuel-library call [1] Regards, Alex [0] https://github.com/stackforge/fuel-main/blob/master/repos.mk#L16-L37 [1] https://github.com/stackforge/fuel-main/blob/master/repos.mk#L45 On Fri, Jul 17, 2015 at 2:37 PM, Alex Schultz aschu...@mirantis.com wrote: Hey Alex, On Jul 17, 2015 4:32 AM, Aleksandr Didenko adide...@mirantis.com wrote: Hi, I think that we should provide a separate script that will fetch the upstream modules into fuel-library/deployment/puppet/ directory. It will allow us to have everything in a single place and use this script in ISO build process and CI jobs. Right. That is what I'm going for. The issue I need help with is the best way to execute this as part of the build process. From what i understand of the build process is that we are using git archive for all pieces so I'm not sure how to wedge in an extra script execution to do the module fetch. The creation of the script isn't the issue, the issue is how can I properly run it as part of the build process. Regards, Alex Thanks, -Alex On Thu, Jul 16, 2015 at 11:17 PM, Alex Schultz aschu...@mirantis.com wrote: Hello everyone, I have committed the initial configuration required to start leveraging librarian-puppet as part of the way we pull in upstream puppet modules[0]. Additionally, I have also committed a change that would pull in the openstack-ironic module[1]. The one piece that is missing from this being a complete solution is the ability to run librarian-puppet as part of our build process for the fuel-library. I've looked into the fuel-main build scripts and I think it's over my head to figure this out just by looking. Can anyone explain to me or assist me in how I could go about modifying the existing build system to be able to run librarian-puppet to prepare the source for the package? In my initial investigation, it looks like it would be a modification of the fuel-main/packages/module.mk[3] file. I basically need to do the prepare_library[3] function from the 202763 review[0] after we've pulled all the sources together to fetch the upstream modules. Thanks, -Alex [0] https://review.openstack.org/202763 [1] https://review.openstack.org/202767 [2] https://github.com/stackforge/fuel-main/blob/master/packages/module.mk#L63-L82 [3] https://review.openstack.org/#/c/202763/1/utils/jenkins/fuel_noop_tests.rb __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe:
Re: [openstack-dev] [fuel] librarian-puppet integration, need help with build tasks for fuel-library
Not until we start using it then any ci that tests with that module will validate the modules inclusion. You can check the output of the jobs as we are printing what modules are managed by librarian. -Alex On Jul 17, 2015 6:17 PM, Andrew Woodward xar...@gmail.com wrote: Fantastic, do we have some way to validate that the module was pulled in properly as part of fuel-library CI? On Fri, Jul 17, 2015 at 2:48 PM Alex Schultz aschu...@mirantis.com wrote: Hey All, I've figured it out without having to modify the fuel-main build code. I've updated the fuel-library spec with a build action that invokes the script to pull down external modules. Please take some time to review the two reviews out there for this change to see if there are any issues with the way it is implemented. https://review.openstack.org/#/c/202763/ https://review.openstack.org/#/c/202767/ This is a first step towards being able to pull in unmodified external puppet modules. Thanks, -Alex On Fri, Jul 17, 2015 at 4:23 PM, Andrew Woodward awoodw...@mirantis.com wrote: On Fri, Jul 17, 2015 at 12:24 PM Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Great that you did this. Now I think I can prepare fuel-main patch to invoke this script right before building fuel-library package. I'll add you to review it. Is it ok if I do this monday morning? Keep in minde we agreeded to a deadline to get this sorted and in shape to land by EOD monday or we will have to retain the old, and crappy fork method. If possible please work out how this needs to work as early as possible so Alex can continue. Vladimir Kozhukalov On Fri, Jul 17, 2015 at 5:51 PM, Alex Schultz aschu...@mirantis.com wrote: Hey Vladimir, On Fri, Jul 17, 2015 at 7:33 AM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Gathering upstream modules certainly should be implemented as a separate script so as to make it possible to use it wherever we need this (tests, builds, etc.) According to builds there are two things 1) We have so called perestroika package build system (Dmitry Burmistrov is a main contributor here). By the end of next week we are going to switch building all the packages to perestroika. And in order to gather upstream modules right before building fuel-library package, we need to change perestroika build scripts. 2) Currently we build packages using make system and you are right about the place where you need to make changes. https://github.com/stackforge/fuel-main/blob/master/packages/module.mk#L63-L82 If you create shell script, I'll help you to add it to make code. I have updated my review[0] to extract the update logic to it's own bash script that can be invoked by the build scripts. Let me know what would be the best way to wedge this in there. I think for the perestroika this would also be needed for the fuel-library build, so if you point me at that I can see if I can help make that change as well. Thanks, -Alex [0] https://review.openstack.org/#/c/202763/ Vladimir Kozhukalov On Fri, Jul 17, 2015 at 2:56 PM, Aleksandr Didenko adide...@mirantis.com wrote: I believe build_repo function is the best way to do this [0]. So for fuel-library we'll need to run a shell script right from the repo before 'touch $$@'. We can make it either conditional ( test -f ./path/additional_build_script.sh bash ./path/additional_build_script.sh ) or as additional parameter to function and add it in fuel-library call [1] Regards, Alex [0] https://github.com/stackforge/fuel-main/blob/master/repos.mk#L16-L37 [1] https://github.com/stackforge/fuel-main/blob/master/repos.mk#L45 On Fri, Jul 17, 2015 at 2:37 PM, Alex Schultz aschu...@mirantis.com wrote: Hey Alex, On Jul 17, 2015 4:32 AM, Aleksandr Didenko adide...@mirantis.com wrote: Hi, I think that we should provide a separate script that will fetch the upstream modules into fuel-library/deployment/puppet/ directory. It will allow us to have everything in a single place and use this script in ISO build process and CI jobs. Right. That is what I'm going for. The issue I need help with is the best way to execute this as part of the build process. From what i understand of the build process is that we are using git archive for all pieces so I'm not sure how to wedge in an extra script execution to do the module fetch. The creation of the script isn't the issue, the issue is how can I properly run it as part of the build process. Regards, Alex Thanks, -Alex On Thu, Jul 16, 2015 at 11:17 PM, Alex Schultz aschu...@mirantis.com wrote: Hello everyone, I have committed the initial configuration required to start leveraging librarian-puppet as part of the way we pull in upstream puppet modules[0]. Additionally, I have also committed a change that would pull in the openstack-ironic module[1]. The one piece that is missing from this being a complete
Re: [openstack-dev] [fuel] librarian-puppet integration, need help with build tasks for fuel-library
On Fri, Jul 17, 2015 at 12:24 PM Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Great that you did this. Now I think I can prepare fuel-main patch to invoke this script right before building fuel-library package. I'll add you to review it. Is it ok if I do this monday morning? Keep in minde we agreeded to a deadline to get this sorted and in shape to land by EOD monday or we will have to retain the old, and crappy fork method. If possible please work out how this needs to work as early as possible so Alex can continue. Vladimir Kozhukalov On Fri, Jul 17, 2015 at 5:51 PM, Alex Schultz aschu...@mirantis.com wrote: Hey Vladimir, On Fri, Jul 17, 2015 at 7:33 AM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Gathering upstream modules certainly should be implemented as a separate script so as to make it possible to use it wherever we need this (tests, builds, etc.) According to builds there are two things 1) We have so called perestroika package build system (Dmitry Burmistrov is a main contributor here). By the end of next week we are going to switch building all the packages to perestroika. And in order to gather upstream modules right before building fuel-library package, we need to change perestroika build scripts. 2) Currently we build packages using make system and you are right about the place where you need to make changes. https://github.com/stackforge/fuel-main/blob/master/packages/module.mk#L63-L82 If you create shell script, I'll help you to add it to make code. I have updated my review[0] to extract the update logic to it's own bash script that can be invoked by the build scripts. Let me know what would be the best way to wedge this in there. I think for the perestroika this would also be needed for the fuel-library build, so if you point me at that I can see if I can help make that change as well. Thanks, -Alex [0] https://review.openstack.org/#/c/202763/ Vladimir Kozhukalov On Fri, Jul 17, 2015 at 2:56 PM, Aleksandr Didenko adide...@mirantis.com wrote: I believe build_repo function is the best way to do this [0]. So for fuel-library we'll need to run a shell script right from the repo before 'touch $$@'. We can make it either conditional ( test -f ./path/additional_build_script.sh bash ./path/additional_build_script.sh ) or as additional parameter to function and add it in fuel-library call [1] Regards, Alex [0] https://github.com/stackforge/fuel-main/blob/master/repos.mk#L16-L37 [1] https://github.com/stackforge/fuel-main/blob/master/repos.mk#L45 On Fri, Jul 17, 2015 at 2:37 PM, Alex Schultz aschu...@mirantis.com wrote: Hey Alex, On Jul 17, 2015 4:32 AM, Aleksandr Didenko adide...@mirantis.com wrote: Hi, I think that we should provide a separate script that will fetch the upstream modules into fuel-library/deployment/puppet/ directory. It will allow us to have everything in a single place and use this script in ISO build process and CI jobs. Right. That is what I'm going for. The issue I need help with is the best way to execute this as part of the build process. From what i understand of the build process is that we are using git archive for all pieces so I'm not sure how to wedge in an extra script execution to do the module fetch. The creation of the script isn't the issue, the issue is how can I properly run it as part of the build process. Regards, Alex Thanks, -Alex On Thu, Jul 16, 2015 at 11:17 PM, Alex Schultz aschu...@mirantis.com wrote: Hello everyone, I have committed the initial configuration required to start leveraging librarian-puppet as part of the way we pull in upstream puppet modules[0]. Additionally, I have also committed a change that would pull in the openstack-ironic module[1]. The one piece that is missing from this being a complete solution is the ability to run librarian-puppet as part of our build process for the fuel-library. I've looked into the fuel-main build scripts and I think it's over my head to figure this out just by looking. Can anyone explain to me or assist me in how I could go about modifying the existing build system to be able to run librarian-puppet to prepare the source for the package? In my initial investigation, it looks like it would be a modification of the fuel-main/packages/module.mk[3] file. I basically need to do the prepare_library[3] function from the 202763 review[0] after we've pulled all the sources together to fetch the upstream modules. Thanks, -Alex [0] https://review.openstack.org/202763 [1] https://review.openstack.org/202767 [2] https://github.com/stackforge/fuel-main/blob/master/packages/module.mk#L63-L82 [3] https://review.openstack.org/#/c/202763/1/utils/jenkins/fuel_noop_tests.rb __ OpenStack Development Mailing List (not for usage questions)
Re: [openstack-dev] [fuel] librarian-puppet integration, need help with build tasks for fuel-library
Hey Vladimir, So I've been playing with it at the moment because I think the better place to do the script execution is as part of the build process controlled by the fuel-library7.0 spec file[0]. It seems to be a valid way to do it (and would work for our CI jobs to) but the issue i'm running into is the use of ruby/bundler to pull in these packages. Any thoughts on the best way to try and provide the required build environment tools since it appears we build on Centos 6.5 at the moment so I'm running into ruby 1.8.7 issues. -Alex [0] https://review.openstack.org/#/c/202763/9/specs/fuel-library7.0.spec On Fri, Jul 17, 2015 at 2:24 PM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Great that you did this. Now I think I can prepare fuel-main patch to invoke this script right before building fuel-library package. I'll add you to review it. Is it ok if I do this monday morning? Vladimir Kozhukalov On Fri, Jul 17, 2015 at 5:51 PM, Alex Schultz aschu...@mirantis.com wrote: Hey Vladimir, On Fri, Jul 17, 2015 at 7:33 AM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Gathering upstream modules certainly should be implemented as a separate script so as to make it possible to use it wherever we need this (tests, builds, etc.) According to builds there are two things 1) We have so called perestroika package build system (Dmitry Burmistrov is a main contributor here). By the end of next week we are going to switch building all the packages to perestroika. And in order to gather upstream modules right before building fuel-library package, we need to change perestroika build scripts. 2) Currently we build packages using make system and you are right about the place where you need to make changes. https://github.com/stackforge/fuel-main/blob/master/packages/module.mk#L63-L82 If you create shell script, I'll help you to add it to make code. I have updated my review[0] to extract the update logic to it's own bash script that can be invoked by the build scripts. Let me know what would be the best way to wedge this in there. I think for the perestroika this would also be needed for the fuel-library build, so if you point me at that I can see if I can help make that change as well. Thanks, -Alex [0] https://review.openstack.org/#/c/202763/ Vladimir Kozhukalov On Fri, Jul 17, 2015 at 2:56 PM, Aleksandr Didenko adide...@mirantis.com wrote: I believe build_repo function is the best way to do this [0]. So for fuel-library we'll need to run a shell script right from the repo before 'touch $$@'. We can make it either conditional ( test -f ./path/additional_build_script.sh bash ./path/additional_build_script.sh ) or as additional parameter to function and add it in fuel-library call [1] Regards, Alex [0] https://github.com/stackforge/fuel-main/blob/master/repos.mk#L16-L37 [1] https://github.com/stackforge/fuel-main/blob/master/repos.mk#L45 On Fri, Jul 17, 2015 at 2:37 PM, Alex Schultz aschu...@mirantis.com wrote: Hey Alex, On Jul 17, 2015 4:32 AM, Aleksandr Didenko adide...@mirantis.com wrote: Hi, I think that we should provide a separate script that will fetch the upstream modules into fuel-library/deployment/puppet/ directory. It will allow us to have everything in a single place and use this script in ISO build process and CI jobs. Right. That is what I'm going for. The issue I need help with is the best way to execute this as part of the build process. From what i understand of the build process is that we are using git archive for all pieces so I'm not sure how to wedge in an extra script execution to do the module fetch. The creation of the script isn't the issue, the issue is how can I properly run it as part of the build process. Regards, Alex Thanks, -Alex On Thu, Jul 16, 2015 at 11:17 PM, Alex Schultz aschu...@mirantis.com wrote: Hello everyone, I have committed the initial configuration required to start leveraging librarian-puppet as part of the way we pull in upstream puppet modules[0]. Additionally, I have also committed a change that would pull in the openstack-ironic module[1]. The one piece that is missing from this being a complete solution is the ability to run librarian-puppet as part of our build process for the fuel-library. I've looked into the fuel-main build scripts and I think it's over my head to figure this out just by looking. Can anyone explain to me or assist me in how I could go about modifying the existing build system to be able to run librarian-puppet to prepare the source for the package? In my initial investigation, it looks like it would be a modification of the fuel-main/packages/module.mk[3] file. I basically need to do the prepare_library[3] function from the 202763 review[0] after we've pulled all the sources together to fetch the upstream modules. Thanks, -Alex [0]
Re: [openstack-dev] [fuel] librarian-puppet integration, need help with build tasks for fuel-library
Hey All, I've figured it out without having to modify the fuel-main build code. I've updated the fuel-library spec with a build action that invokes the script to pull down external modules. Please take some time to review the two reviews out there for this change to see if there are any issues with the way it is implemented. https://review.openstack.org/#/c/202763/ https://review.openstack.org/#/c/202767/ This is a first step towards being able to pull in unmodified external puppet modules. Thanks, -Alex On Fri, Jul 17, 2015 at 4:23 PM, Andrew Woodward awoodw...@mirantis.com wrote: On Fri, Jul 17, 2015 at 12:24 PM Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Great that you did this. Now I think I can prepare fuel-main patch to invoke this script right before building fuel-library package. I'll add you to review it. Is it ok if I do this monday morning? Keep in minde we agreeded to a deadline to get this sorted and in shape to land by EOD monday or we will have to retain the old, and crappy fork method. If possible please work out how this needs to work as early as possible so Alex can continue. Vladimir Kozhukalov On Fri, Jul 17, 2015 at 5:51 PM, Alex Schultz aschu...@mirantis.com wrote: Hey Vladimir, On Fri, Jul 17, 2015 at 7:33 AM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Gathering upstream modules certainly should be implemented as a separate script so as to make it possible to use it wherever we need this (tests, builds, etc.) According to builds there are two things 1) We have so called perestroika package build system (Dmitry Burmistrov is a main contributor here). By the end of next week we are going to switch building all the packages to perestroika. And in order to gather upstream modules right before building fuel-library package, we need to change perestroika build scripts. 2) Currently we build packages using make system and you are right about the place where you need to make changes. https://github.com/stackforge/fuel-main/blob/master/packages/module.mk#L63-L82 If you create shell script, I'll help you to add it to make code. I have updated my review[0] to extract the update logic to it's own bash script that can be invoked by the build scripts. Let me know what would be the best way to wedge this in there. I think for the perestroika this would also be needed for the fuel-library build, so if you point me at that I can see if I can help make that change as well. Thanks, -Alex [0] https://review.openstack.org/#/c/202763/ Vladimir Kozhukalov On Fri, Jul 17, 2015 at 2:56 PM, Aleksandr Didenko adide...@mirantis.com wrote: I believe build_repo function is the best way to do this [0]. So for fuel-library we'll need to run a shell script right from the repo before 'touch $$@'. We can make it either conditional ( test -f ./path/additional_build_script.sh bash ./path/additional_build_script.sh ) or as additional parameter to function and add it in fuel-library call [1] Regards, Alex [0] https://github.com/stackforge/fuel-main/blob/master/repos.mk#L16-L37 [1] https://github.com/stackforge/fuel-main/blob/master/repos.mk#L45 On Fri, Jul 17, 2015 at 2:37 PM, Alex Schultz aschu...@mirantis.com wrote: Hey Alex, On Jul 17, 2015 4:32 AM, Aleksandr Didenko adide...@mirantis.com wrote: Hi, I think that we should provide a separate script that will fetch the upstream modules into fuel-library/deployment/puppet/ directory. It will allow us to have everything in a single place and use this script in ISO build process and CI jobs. Right. That is what I'm going for. The issue I need help with is the best way to execute this as part of the build process. From what i understand of the build process is that we are using git archive for all pieces so I'm not sure how to wedge in an extra script execution to do the module fetch. The creation of the script isn't the issue, the issue is how can I properly run it as part of the build process. Regards, Alex Thanks, -Alex On Thu, Jul 16, 2015 at 11:17 PM, Alex Schultz aschu...@mirantis.com wrote: Hello everyone, I have committed the initial configuration required to start leveraging librarian-puppet as part of the way we pull in upstream puppet modules[0]. Additionally, I have also committed a change that would pull in the openstack-ironic module[1]. The one piece that is missing from this being a complete solution is the ability to run librarian-puppet as part of our build process for the fuel-library. I've looked into the fuel-main build scripts and I think it's over my head to figure this out just by looking. Can anyone explain to me or assist me in how I could go about modifying the existing build system to be able to run librarian-puppet to prepare the source for the package? In my initial investigation, it looks like it would be a modification of the
Re: [openstack-dev] [fuel] librarian-puppet integration, need help with build tasks for fuel-library
Fantastic, do we have some way to validate that the module was pulled in properly as part of fuel-library CI? On Fri, Jul 17, 2015 at 2:48 PM Alex Schultz aschu...@mirantis.com wrote: Hey All, I've figured it out without having to modify the fuel-main build code. I've updated the fuel-library spec with a build action that invokes the script to pull down external modules. Please take some time to review the two reviews out there for this change to see if there are any issues with the way it is implemented. https://review.openstack.org/#/c/202763/ https://review.openstack.org/#/c/202767/ This is a first step towards being able to pull in unmodified external puppet modules. Thanks, -Alex On Fri, Jul 17, 2015 at 4:23 PM, Andrew Woodward awoodw...@mirantis.com wrote: On Fri, Jul 17, 2015 at 12:24 PM Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Great that you did this. Now I think I can prepare fuel-main patch to invoke this script right before building fuel-library package. I'll add you to review it. Is it ok if I do this monday morning? Keep in minde we agreeded to a deadline to get this sorted and in shape to land by EOD monday or we will have to retain the old, and crappy fork method. If possible please work out how this needs to work as early as possible so Alex can continue. Vladimir Kozhukalov On Fri, Jul 17, 2015 at 5:51 PM, Alex Schultz aschu...@mirantis.com wrote: Hey Vladimir, On Fri, Jul 17, 2015 at 7:33 AM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Alex, Gathering upstream modules certainly should be implemented as a separate script so as to make it possible to use it wherever we need this (tests, builds, etc.) According to builds there are two things 1) We have so called perestroika package build system (Dmitry Burmistrov is a main contributor here). By the end of next week we are going to switch building all the packages to perestroika. And in order to gather upstream modules right before building fuel-library package, we need to change perestroika build scripts. 2) Currently we build packages using make system and you are right about the place where you need to make changes. https://github.com/stackforge/fuel-main/blob/master/packages/module.mk#L63-L82 If you create shell script, I'll help you to add it to make code. I have updated my review[0] to extract the update logic to it's own bash script that can be invoked by the build scripts. Let me know what would be the best way to wedge this in there. I think for the perestroika this would also be needed for the fuel-library build, so if you point me at that I can see if I can help make that change as well. Thanks, -Alex [0] https://review.openstack.org/#/c/202763/ Vladimir Kozhukalov On Fri, Jul 17, 2015 at 2:56 PM, Aleksandr Didenko adide...@mirantis.com wrote: I believe build_repo function is the best way to do this [0]. So for fuel-library we'll need to run a shell script right from the repo before 'touch $$@'. We can make it either conditional ( test -f ./path/additional_build_script.sh bash ./path/additional_build_script.sh ) or as additional parameter to function and add it in fuel-library call [1] Regards, Alex [0] https://github.com/stackforge/fuel-main/blob/master/repos.mk#L16-L37 [1] https://github.com/stackforge/fuel-main/blob/master/repos.mk#L45 On Fri, Jul 17, 2015 at 2:37 PM, Alex Schultz aschu...@mirantis.com wrote: Hey Alex, On Jul 17, 2015 4:32 AM, Aleksandr Didenko adide...@mirantis.com wrote: Hi, I think that we should provide a separate script that will fetch the upstream modules into fuel-library/deployment/puppet/ directory. It will allow us to have everything in a single place and use this script in ISO build process and CI jobs. Right. That is what I'm going for. The issue I need help with is the best way to execute this as part of the build process. From what i understand of the build process is that we are using git archive for all pieces so I'm not sure how to wedge in an extra script execution to do the module fetch. The creation of the script isn't the issue, the issue is how can I properly run it as part of the build process. Regards, Alex Thanks, -Alex On Thu, Jul 16, 2015 at 11:17 PM, Alex Schultz aschu...@mirantis.com wrote: Hello everyone, I have committed the initial configuration required to start leveraging librarian-puppet as part of the way we pull in upstream puppet modules[0]. Additionally, I have also committed a change that would pull in the openstack-ironic module[1]. The one piece that is missing from this being a complete solution is the ability to run librarian-puppet as part of our build process for the fuel-library. I've looked into the fuel-main build scripts and I think it's over my head to figure this out just by looking. Can anyone explain to me or assist me in how I could go about
