What if we implement plugins not as separate Puppet modules, but as separate Mcollective modules?
On Thu, Jan 16, 2014 at 3:43 PM, Dmitry Ilyin <[email protected]> wrote: > Actually you have risen a very complicated question... > > We have been thinking about separating our deployment process for a long > time already but have not done anything so far. > Currently we are using a single 'main' manifest that is applied to every > node. It uses puppet modules as 'library' and data from astute.yaml. > This huge manifest is far too large to be easily understood and is full of > hacks and kludges. > It makes reference architectures very unflexible and custom roles hard to > implement. > And I'm not even talking about joys of debugging all this stuff and a lot > of dependency and ordering fun we have with it. > Adding something new requires not only deep understanding of this > structure but also a lot of luck not to break something accidently. > It makes pretty much impossible for an external developer to write a > plugin or custom role he could just add to Fuel an have it working as > expected. > > So we have an idea to separate this manifest to many small ones. Each of > them would do only one task and orchestration engine would apply them in > required order > to deploy the node user have requested. With such architecture each > 'stage' would be more or less isolated and it would be possible for a > single developer > to add a new 'stage' that would install his 'plugin'. > > Yes, Dmitry's idea about implementing such plugins as separate puppet > modules is good, but it's not that easy to do. > > * Modules can contain their own manifests, files and templates and can use > configuration from astute.yaml. It will makes possible to represent every > service we install as plugin and every 'role' would be a set of > 'plugins' installed in certain order. Orchestrator will have to run 'puppet > apply' on each > module's init.pp to get the job done. > * Separating single puppet run to many runs will let us to bypass resource > uniqness restriction and we'll be able to manage same resources on > different 'stages' without hacks to prevent conflicts. > * Some 'stages' could even be done not by puppet if they are really hard > to do by puppet. > > But there are questins I have no ansver yet. > > * Even if we'll make each 'stage' a separate manifest there still we be a > need to use our current puppet modules as library. Or we should make every > modules fully self-contained? Or we could use something like python's > virtualenv to provide each 'stage' with correct set of library modules? > * What is the best way to install packages then? Keep our current > repository on master node or invent somethin else? > * How and where to store information on what 'stages' should be done for > which roles? There should be some metadata with fixed scenarios or there > should be dynamic graph? > * Some 'plugins' and 'stages' may require to add options to settings > panel. If we are going to have external plugins there should be a way to > hook int setting and add your options. > This data will the be available in astute.yaml. > * There should be some system to run pre-deploy and post-deploy tests for > each stage to determine was it successful. > * Our orchestation engine will have to become much more complex then what > we have now. There should be queues of stages for each deploying node. And > there should be support > for inter-node dependencies of stages. > > I'm working new architecture blueprint. I'll send updates when i'll be > ready. > > > > > > 2013/12/28 Roman Alekseenkov <[email protected]> > >> Dmitry - sounds like a great start. >> >> Team, >> >> Pluggable architecture is a priority for 4.1. Given that we are now done >> with Havana release, please make an effort to review and give prompt >> feedback to Dmitry. >> >> Thanks, >> Roman >> >> >> On Wednesday, December 25, 2013, Dmitry Ukov wrote: >> >>> Hello folks, >>> recently I've refactored integration of Neutron Nicira NSX plugin >>> installation into fuel library (https://review.openstack.org/#/c/64002/). >>> All modifications are gathered into one puppet module. >>> I have some thoughts on implementation of plugin system in Fuel library >>> >>> 1. Each plugin has to be implemented as a separate module. Init >>> manifest withouta input parameters (init.pp) should be used as an entry >>> point to each plugin. >>> 2. All parameters for plugin should be kept in astute.yaml >>> 3. Plugin naming convention should be developed. >>> 4. Function which automatically loads each plugin (e.g. it should >>> execute init manifest of each class with name mask "plugin_.*") is >>> required. >>> 5. Staging system should be implemented. All plugins have to be >>> executed in a separate stage (e.g. Stage['main'] -> stage {'plugins':} ) >>> 6. Each puppet class which is used with library has to have the >>> ability to be disabled from plugin >>> This is necessary because plugins may require to disable >>> some services. Alternatively this can be implemented by using resource >>> collections (e.g. Service <| title == l3-agent|> { noop => true }) but >>> we can't execute plugin in a separate stage in this case. >>> >>> What do you think on this? I'd like hear a feedback on my ideas and >>> module implementation. >>> Thank you in advance. >>> >>> -- >>> Kind regards >>> Dmitry Ukov >>> IT Engineer >>> Mirantis, Inc. >>> >>> >> -- >> Mailing list: https://launchpad.net/~fuel-dev >> Post to : [email protected] >> Unsubscribe : https://launchpad.net/~fuel-dev >> More help : https://help.launchpad.net/ListHelp >> >> > > -- > Mailing list: https://launchpad.net/~fuel-dev > Post to : [email protected] > Unsubscribe : https://launchpad.net/~fuel-dev > More help : https://help.launchpad.net/ListHelp > >
-- Mailing list: https://launchpad.net/~fuel-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~fuel-dev More help : https://help.launchpad.net/ListHelp
