We could still use Puppet, just run different sets of modules at different stages, for instance.
-Oleg On Thu, Jan 16, 2014 at 5:42 PM, Dmitry Ilyin <[email protected]> wrote: > Yes, it's possible to write mcollective plugin to execute each deployment > stage and then use orchestration to run them. > But mcollective is not very good at running long actions. > We are using it to run puppet but it instantly forks and exits and then we > use lock file and status file to get status of deployment. > We'll keep using mcollective but most likely it will require a new agent > which would be able to run plugins and get their deploymnet status. > If all our plugins will be puppet manifests we can continue to use our > current puppet mco agent and just pass the name of the manifest we want to > apply. > > > 2014/1/16 Oleg Gelbukh <[email protected]> > >> What if we implement plugins not as separate Puppet modules, but as >> separate Mcollective modules? >> >> >> On Thu, Jan 16, 2014 at 3:43 PM, Dmitry Ilyin <[email protected]>wrote: >> >>> Actually you have risen a very complicated question... >>> >>> We have been thinking about separating our deployment process for a long >>> time already but have not done anything so far. >>> Currently we are using a single 'main' manifest that is applied to every >>> node. It uses puppet modules as 'library' and data from astute.yaml. >>> This huge manifest is far too large to be easily understood and is full >>> of hacks and kludges. >>> It makes reference architectures very unflexible and custom roles hard >>> to implement. >>> And I'm not even talking about joys of debugging all this stuff and a >>> lot of dependency and ordering fun we have with it. >>> Adding something new requires not only deep understanding of this >>> structure but also a lot of luck not to break something accidently. >>> It makes pretty much impossible for an external developer to write a >>> plugin or custom role he could just add to Fuel an have it working as >>> expected. >>> >>> So we have an idea to separate this manifest to many small ones. Each of >>> them would do only one task and orchestration engine would apply them in >>> required order >>> to deploy the node user have requested. With such architecture each >>> 'stage' would be more or less isolated and it would be possible for a >>> single developer >>> to add a new 'stage' that would install his 'plugin'. >>> >>> Yes, Dmitry's idea about implementing such plugins as separate puppet >>> modules is good, but it's not that easy to do. >>> >>> * Modules can contain their own manifests, files and templates and can >>> use configuration from astute.yaml. It will makes possible to represent >>> every >>> service we install as plugin and every 'role' would be a set of >>> 'plugins' installed in certain order. Orchestrator will have to run 'puppet >>> apply' on each >>> module's init.pp to get the job done. >>> * Separating single puppet run to many runs will let us to bypass >>> resource uniqness restriction and we'll be able to manage same resources on >>> different 'stages' without hacks to prevent conflicts. >>> * Some 'stages' could even be done not by puppet if they are really hard >>> to do by puppet. >>> >>> But there are questins I have no ansver yet. >>> >>> * Even if we'll make each 'stage' a separate manifest there still we be >>> a need to use our current puppet modules as library. Or we should make every >>> modules fully self-contained? Or we could use something like python's >>> virtualenv to provide each 'stage' with correct set of library modules? >>> * What is the best way to install packages then? Keep our current >>> repository on master node or invent somethin else? >>> * How and where to store information on what 'stages' should be done for >>> which roles? There should be some metadata with fixed scenarios or there >>> should be dynamic graph? >>> * Some 'plugins' and 'stages' may require to add options to settings >>> panel. If we are going to have external plugins there should be a way to >>> hook int setting and add your options. >>> This data will the be available in astute.yaml. >>> * There should be some system to run pre-deploy and post-deploy tests >>> for each stage to determine was it successful. >>> * Our orchestation engine will have to become much more complex then >>> what we have now. There should be queues of stages for each deploying node. >>> And there should be support >>> for inter-node dependencies of stages. >>> >>> I'm working new architecture blueprint. I'll send updates when i'll be >>> ready. >>> >>> >>> >>> >>> >>> 2013/12/28 Roman Alekseenkov <[email protected]> >>> >>>> Dmitry - sounds like a great start. >>>> >>>> Team, >>>> >>>> Pluggable architecture is a priority for 4.1. Given that we are now >>>> done with Havana release, please make an effort to review and give prompt >>>> feedback to Dmitry. >>>> >>>> Thanks, >>>> Roman >>>> >>>> >>>> On Wednesday, December 25, 2013, Dmitry Ukov wrote: >>>> >>>>> Hello folks, >>>>> recently I've refactored integration of Neutron Nicira NSX plugin >>>>> installation into fuel library ( >>>>> https://review.openstack.org/#/c/64002/). All modifications are >>>>> gathered into one puppet module. >>>>> I have some thoughts on implementation of plugin system in Fuel library >>>>> >>>>> 1. Each plugin has to be implemented as a separate module. Init >>>>> manifest withouta input parameters (init.pp) should be used as an entry >>>>> point to each plugin. >>>>> 2. All parameters for plugin should be kept in astute.yaml >>>>> 3. Plugin naming convention should be developed. >>>>> 4. Function which automatically loads each plugin (e.g. it should >>>>> execute init manifest of each class with name mask "plugin_.*") is >>>>> required. >>>>> 5. Staging system should be implemented. All plugins have to be >>>>> executed in a separate stage (e.g. Stage['main'] -> stage {'plugins':} >>>>> ) >>>>> 6. Each puppet class which is used with library has to have the >>>>> ability to be disabled from plugin >>>>> This is necessary because plugins may require to disable >>>>> some services. Alternatively this can be implemented by using resource >>>>> collections (e.g. Service <| title == l3-agent|> { noop => true }) but >>>>> we can't execute plugin in a separate stage in this case. >>>>> >>>>> What do you think on this? I'd like hear a feedback on my ideas and >>>>> module implementation. >>>>> Thank you in advance. >>>>> >>>>> -- >>>>> Kind regards >>>>> Dmitry Ukov >>>>> IT Engineer >>>>> Mirantis, Inc. >>>>> >>>>> >>>> -- >>>> Mailing list: https://launchpad.net/~fuel-dev >>>> Post to : [email protected] >>>> Unsubscribe : https://launchpad.net/~fuel-dev >>>> More help : https://help.launchpad.net/ListHelp >>>> >>>> >>> >>> -- >>> Mailing list: https://launchpad.net/~fuel-dev >>> Post to : [email protected] >>> Unsubscribe : https://launchpad.net/~fuel-dev >>> More help : https://help.launchpad.net/ListHelp >>> >>> >> >
-- Mailing list: https://launchpad.net/~fuel-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~fuel-dev More help : https://help.launchpad.net/ListHelp
