[openstack-dev] [Fuel] Help with reviews - modularisation.
Hi Fuelers! I kindly ask you to review my patches [0] and [1]. There is a "big thing" going on there. I'm trying to decouple Volume Manager from Nailgun. It's a first step towards Fuel modularisation, which will also be an example for other developers: "How to implement Nailgun extensions". There are many places in the code which can be tricky. I'm not able to see all cases or all potential problems and I'm asking you guys to share your opinion. [0] https://review.openstack.org/#/c/291763/ - Extract volume manager [1] https://review.openstack.org/#/c/294510/ - Some of the more important tests are also moved out instead of deleting them. Regards -- *Sylwester Brzeczkowski* Python Software Engineer Product Development-Core : Product Engineering __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Infra] Cross-repository testing
Jeremy: Thanks for your response! So the solution I proposed fits in common openstack practices, that's cool. I will definitely use zuul-cloner as it works for this case, but I would use it inside our script and then invoke the script in job macro. That will ensure that we will have the same mechanism working on local machine and on CI. What do you think? Regards On Thu, Mar 24, 2016 at 2:50 PM, Jeremy Stanley wrote: > On 2016-03-24 11:44:16 +0100 (+0100), Sylwester Brzeczkowski wrote: > > We want to enable proper testing for Nailgun (fuel-web) extensions. We > want > > to implement a script which will clone all required repos for test and > > actually run the tests. > > > > The script will be placed in openstack/fuel-web [0] repo and should work > > locally and also on openstack CI (gate jobs). All Nailgun Extensions > should > > have Nailgun in it's requirements so I think it's ok. > > > > What do you think about the idea? Is it a good approach? > > Am I missing some already existing solutions for this problem? > [...] > > This is basically what devstack-gate does for testing OpenStack > deployments with DevStack. The logic for getting change dependencies > right is far more complex than simply cloning a bunch of > repositories. You need to make sure you try to fetch appropriate Zuul > refs for all of them when available, fallback to sane branch > selections when not all of the repos have the same branches, make > use of repo caches for improved performance and update them to > current state from our remote mirror, et cetera. > > Zuul includes a standalone command-line tool which also performs > these operations, and we make it available on all our job workers at > /usr/zuul-env/bin/zuul-cloner. Check job macros in the > project-config directory for numerous examples of creating clonemaps > and invoking zuul-cloner in jobs. > -- > Jeremy Stanley > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- *Sylwester Brzeczkowski* Python Software Engineer Product Development-Core : Product Engineering __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Infra] Cross-repository testing
Hi all! We want to enable proper testing for Nailgun (fuel-web) extensions. We want to implement a script which will clone all required repos for test and actually run the tests. The script will be placed in openstack/fuel-web [0] repo and should work locally and also on openstack CI (gate jobs). All Nailgun Extensions should have Nailgun in it's requirements so I think it's ok. What do you think about the idea? Is it a good approach? Am I missing some already existing solutions for this problem? PS: I have sent 1 email about this already [1], but I decided to change the audience to broader one since question is not Fuel-specific it's more about asking for feedback. Please read the email if you're interested in details. [0] https://github.com/openstack/fuel-web [1] http://lists.openstack.org/pipermail/openstack-dev/2016-March/089660.html -- *Sylwester Brzeczkowski* Python Software Engineer Product Development-Core : Product Engineering __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Fuel][Infra] Nailgun extensions testing
Hi everyone! I’m looking for boilerplates/good practices regarding to testing extensions with core code. Since we unlocked Nailgun extensions system [0] and now there is a possibility to install the extensions from external sources we want to also provide a way to test your own extensions against Nailgun and some other extensions. Here is the spec for this activity [1] The idea is to write python (or shell) script which will: - clone all required repos (like fuel-web, extensions repos) using probably zuul-cloner - checkout to appropriate stable branches / will cherry-pick some commit / stay on master - run tests This script will be used to: - test extension with different Nailgun versions (to check if it’s compatible) locally and on extension’s jenkins gate jobs - test extension with different Nailgun versions and with other extensions enabled (depending on needs) - test Nailgun with some core extensions locally and on fuel-web jenkins gate jobs The script will be placed in fuel-web repo as extensions will need to have Nailgun in its requirements anyway. There will be new jenkins job which will consume names of extensions to test and the branches/commits/versions which the tests should be run against. The job will basically fetch fuel-web repo, and run the script mentioned above. What do you think about the idea? Is it a good approach? Am I missing some already existing solutions for this problem? Regards [0] https://blueprints.launchpad.net/fuel/+spec/stevedore-extensions-discovery [1] https://review.openstack.org/#/c/281749/ -- *Sylwester Brzeczkowski* Python Software Engineer Product Development-Core : Product Engineering __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Rewriting nailgun agent on Python proposal
+1 to drop nailgun-agent and replace it with python script with ohai call or ironic-inspector! On Tue, Mar 15, 2016 at 1:07 PM, Alexander Saprykin wrote: > Dear all, > > Thank you for the opinions about this problem. > > I would agree with Roman, that it is always better to reuse solutions than > re-inventing the wheel. We should investigate possibility of using > ironic-inspector and integrating it into fuel. > > Best regards, > Alexander Saprykin > > 2016-03-15 13:03 GMT+01:00 Sergii Golovatiuk : > >> My strong +1 to drop off nailgun-agent completely in favour of >> ironic-inspector. Even taking into consideration we'lll need to >> extend ironic-inspector for fuel needs. >> >> -- >> Best regards, >> Sergii Golovatiuk, >> Skype #golserge >> IRC #holser >> >> On Tue, Mar 15, 2016 at 11:06 AM, Roman Prykhodchenko >> wrote: >> >>> My opition on this is that we have too many re-invented wheels in Fuel >>> and it’s better think about replacing them with something we can re-use >>> than re-inventing them one more time. >>> >>> Let’s take a look at Ironic and try to figure out how we can use its >>> features for the same purpose. >>> >>> >>> - romcheg >>> > 15 бер. 2016 р. о 10:38 Neil Jerram >>> написав(ла): >>> > >>> > On 15/03/16 07:11, Vladimir Kozhukalov wrote: >>> >> Alexander, >>> >> >>> >> We have many other places where use Ruby (astute, puppet custom types, >>> >> etc.). I don't think it is a good reason to re-write something just >>> >> because it is written in Ruby. You are right about tests, about >>> plugins, >>> >> but let's look around. Ironic community has already invented discovery >>> >> component (btw written in python) and I can't see any reason why we >>> >> should continue putting efforts in nailgun agent and not try to switch >>> >> to ironic-inspector. >>> > >>> > +1 in general terms. It's strange to me that there are so many >>> > OpenStack deployment systems that each do each piece of the puzzle in >>> > their own way (Fuel, Foreman, MAAS/Juju etc.) - and which also means >>> > that I need substantial separate learning in order to use all these >>> > systems. It would be great to see some consolidation. >>> > >>> > Regards, >>> > Neil >>> > >>> > >>> > >>> __ >>> > OpenStack Development Mailing List (not for usage questions) >>> > Unsubscribe: >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >>> >>> __ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- *Sylwester Brzeczkowski* Python Software Engineer Product Development-Core : Product Engineering __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel][FFE] provisioning and deployment data pipeline
I'm glad to inform you that FFE is no longer needed here. The pipeline implementation was finally merged. I also stated working on fixing random failing test [0]. Regards [0] https://review.openstack.org/#/c/286583/ On Tue, Mar 1, 2016 at 12:38 PM, Sylwester Brzeczkowski < sbrzeczkow...@mirantis.com> wrote: > Hi > > I want to request a feature freeze exception for "provisioning and > deployment data pipeline" > > There is one patch with implementation [1] which is ready and already has > had two '+2's and '+1' workflow yesterday but there still failing gate jobs > because of some random test failures. > > Merging it is a matter of investigation this random failures or keep > 'reverifying' it on gerrit, which I'm now doing simultaneously. > > Regards > > [0] https://blueprints.launchpad.net/fuel/+spec/data-pipeline > [1] https://review.openstack.org/#/c/272977/ > > -- > *Sylwester Brzeczkowski* > Junior Python Software Engineer > Product Development-Core : Product Engineering > -- *Sylwester Brzeczkowski* Junior Python Software Engineer Product Development-Core : Product Engineering __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Fuel][FFE] provisioning and deployment data pipeline
Hi I want to request a feature freeze exception for "provisioning and deployment data pipeline" There is one patch with implementation [1] which is ready and already has had two '+2's and '+1' workflow yesterday but there still failing gate jobs because of some random test failures. Merging it is a matter of investigation this random failures or keep 'reverifying' it on gerrit, which I'm now doing simultaneously. Regards [0] https://blueprints.launchpad.net/fuel/+spec/data-pipeline [1] https://review.openstack.org/#/c/272977/ -- *Sylwester Brzeczkowski* Junior Python Software Engineer Product Development-Core : Product Engineering __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Bareon][Fuel] Stevedore extensions system for Nailgun
Hi, Recently I was working on making Nailgun extensions more usable. Current extensions system has great potential to be used to extend Nailgun's features e.g. integration with other services. This potential is blocked by the fact that all extensions must be placed in Nailgun's source code and be explicitly imported and added to global "extensions list" [0]. There is the idea to use stevedore to unblock current extensions system. Stevedore allows us to create an extension in separate package and make it available just by installing it on Fuel Master node. Nailgun will automatically detect that new extension is ready for use. User will be able to turn on the extension via fuelclient [1] or directly via Nailgun's REST API [2]. I've prepared a Demo [3] in which I show how to use extensions using integration with Bareon-API [4] as an example. The extension is placed in separate repo [5]. Please look at the spec for stevedore-based extensions for Nailgun [6] and review it as soon as possible to move the initiative forward. Any feedback is welcome! Thank you! [0] https://github.com/openstack/fuel-web/blob/stable/8.0/nailgun/nailgun/extensions/base.py#L23-L36 [1] https://review.openstack.org/#/c/264699/ [2] https://review.openstack.org/#/c/264714/ [3] https://www.youtube.com/watch?v=8r0yaoPWtlY [4] https://review.openstack.org/#/c/259072/ [5] https://github.com/gitfred/bareon-fuel-extension [6] https://review.openstack.org/#/c/263738/ -- *Sylwester Brzeczkowski* Junior Python Software Engineer Product Development-Core : Product Engineering __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Patch size limit
Neil, just to clarify: moved/renamed files are marked as "R" so I think there may be some way to ignore such files when counting LOC. Maciej, I completely agree with you. It's pretty hard to review such big change,and takes a lot of time which could be saved by submitting smaller patches. +1 On Tue, Dec 1, 2015 at 1:50 PM, Neil Jerram wrote: > On 01/12/15 12:45, Maciej Kwiek wrote: > > Hi, > > > > I recently noticed the influx of big patches hitting Gerrit > > (especially in fuel-web, but I also heard that there was a couple of > > big ones in library). I think that patches that have 1000 LOC are > > simply too big to review thoroughly and reliably. > > > > I would argue that there should be a limit to patch size, either a > > soft one (i.e. written down in contributor guidelines) or a hard one > > (e.g. enforced by gate job). > > > > I think that we need a discussion whether we really need this limit, > > what should it be, and how to enforce it. > > > > I personally think that most patches that are over 400 LOC could be > > easily split into at least two smaller changes. > > > > Regarding the limit enforcement - I think we should go with the soft > > limit, with X LOC written as a guideline and giving the reviewers a > > possibility to give -1s to patches that are too big, but also giving > > the possibility to merge bigger changes if it's absolutely necessary > > (in really rare cases the changes simply cannot be split). We may mix > > in the hard limit for ridiculously large patches (twice the "soft > > limit" would be good in my opinion), so the gate would automatically > > reject such patches, forcing contributor to split his patch. > > > > Please share your thoughts on this. > > I think most of your principle is correct. However I can imagine a file > renaming / moving patch that would appear in Gerrit to be >=1000 LOC, > but would actually just be file moves, with perhaps some trivial changes > to Python module paths; and I don't think it would be helpful to force a > patch like that to be split up. So it may not be correct to enforce a > hard limit all the time. > > Neil > > > ______ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- *Sylwester Brzeczkowski* Junior Python Software Engineer Product Development-Core : Product Engineering __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Fuel] Infiniband support in Fuel
We have a bug in fuel [1] which concerns infiniband support. It's mostly about expanding the size of "mac" field in db from 17 to 59. But I've email Aviram Bar-Haim who was working on for infiniband support [2] for fuel-plugin-mellanox [3] and he said that they use eIPoIB mac (mapping between ETH to IB) instead of IB Guid so it fits to our current mac field size. Does anyone have ever used fuel-plugin-mellanox with IB? I'm trying to find out if the bug is still valid? [1] https://bugs.launchpad.net/fuel/+bug/1398882 [2] https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/db/sqlalchemy/models/node.py#L103 [3] https://github.com/stackforge/fuel-plugin-mellanox Regards -- *Sylwester Brzeczkowski* Junior Python Software Engineer Product Development-Core : Product Engineering __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
