Re: [Puppet Users] Re: Rethinking Puppet modules
On Tuesday, June 24, 2014 4:20:33 PM UTC+2, Felix.Frank wrote: Hi guys, On 06/24/2014 03:48 PM, jcbollinger wrote: I understand the point, but at the same time I find it harmful also to have hundreds of modules, that are published on the forge but work only for a specific case and are practically useless for others. I disagree. It is not at all a problem -- in itself -- to have hundreds of narrowly-scoped modules on the forge. More modules means more chances to find something that serves my needs, or something I can learn from. The concern is presumably about how easy or hard it is to find 'the' right module for me, but I don't think the solution is to try to ensure that all modules published there meet specific, high API and functionality criteria. Rather, the solution is the same as ever in an open-source community: documentation, reputation (both of the code and of the author), popularity (mostly of the code), and support and recommendations from the community, especially (but not exclusively) from thought leaders. I daresay, for example, that many people choose /your/ modules in large part just because they are yours. You and your modules have strong reputations, which are reinforced when users find that your modules work well for them (or undermined in the unlikely event that they find otherwise). Perhaps the Forge platform could be enhanced to make those tools easier to apply in its context, but at least some of them are already reasonably accessible. I have followed this thread with great interest, and although I didn't take the time to read the blog posts, I have to say that I'm pretty much 100% with Alessandro on this one. My own dives into the Forge have been short-lived so far, but the current experience is detrimental, I feel. The multitude of solutions and the fact that there may actually be two or more modules that each suit *some* specific needs of mine, with no visible impetus towards merging similar modules into a one fits all solution makes me despair early. It is my fear that many potential users are repelled by the same experience. It would be my hope that each problem that modules solve could culminate in one module that strikes just the right balance between being generic enough to serve (most) everyone without being overladen with features that almost nobody needs. (Such go-to modules would likely be taken under the umbrella of supported modules.) That is not to say that there should be no ecosystem, we just cannot have too many dead leaves and branches scattered everywhere. Those should be recycled into better core modules by some means. If I read him correctly, Alessandro is currently questing for a way to make just this unification process possible, although that's possibly just my interpretation. (But a common set of practices *will* make the merging of functionality much easier.) I support this. Cool. Any contribution to https://github.com/example42/puppet-tp is welcomed. I'm currently figuring out what's the best way to organise and structure the module data and have to fix the tp_lookup function that is the core of the hiera-like retrieval mechanism. Once these basic parts are fixed I will start to populate aggressively the data directory (with data for different applications) and then take care of the puppet tp face. Anyway , just to make it clearer, the intended usage of tp is to ease and standardise the management of software that can be installed with normal native packages (eventually providing a custom repo via the dependency_class parameter). It's not and will never be a complete replacement of other modules, rather I think of it as a complementary tool, to be used instead of full modules for some cases (where you deal with just packages , services and files resources) or together with an existing module (which might provide some application specific resources (such as mysql::grant) that a generic tp module will never have) in other cases. It's intended to be used in higher abstraction modules where can stay the logic of how different resources are grouped ad configured. Think of it as the tool that can let us define the shape of the single lego pieces, but these pieces have to be assembled together somewhere else. al -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/e855d2fe-e159-4579-96a9-6a3e9dbf9263%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Re: Rethinking Puppet modules
On Tuesday, June 24, 2014 4:27:41 PM UTC+2, pablo.f...@cscs.ch wrote: Hi, On 23/06/14 19:15, Alessandro Franceschi wrote: As a side note I'm a bit perplex about the proliferation of parameters in classes or defines in order to match single application specific configuration entries. When I look at this, for example: https://github.com/puppetlabs/puppetlabs-apache/blob/master/manifests/vhost.pp my backbone freezes. A huge amount of define parameters, that map Apache directives, and not even a parameter that allows me to provide a custom template. I agree... it sucks! But I think it sucks more to the person that wrote the module than to those using it. When you see a declaration of a vhost, written in puppet language, it's clear and self-explanatory. But if you pass a template instead, the clarity is lost: you need to look for it elsewhere, it's much more complicated to understand a template than a puppet hash, and you need to seek each variable to see where its value is coming from. The specific problem with vhosts (and other of the kind) is a problem with the language itself: Hiera does not search parameters of defined types, and you need to find your way around with a class that does a simple create_resources to create those resources of that defined type (and probably set defaults with parameters). It could be worse, though :) Also, if you want a pure data driven setup, where you place on hiera hashes of data with all the stuff you need, you can still have this using a template and a single parameters ( such as the options_hash ) I have mixed feelings about this: if it's not a parameter, you don't have automatic hiera lookups, and additionally you have to properly document the structure of the hash. Parameters are almost self-explanatory... but I agree that there are some modules where everything is a parameter, and it's difficult to handle. But I think I would rather have that than doing a template do all the work and have to parameters (template and options_hash) Sometimes I wonder if a totally data driven approach is something that is always the right choice. Given that it's mostly a matter of personal preference, infrastructure design, skills and so on, I feel like that sometimes modules' author forget that Puppet users are most of the times system administrators that know exactly how to configure their files and need Puppet to replicate the configurations they have. So having to study a module in order to achieve the intended output on the generated configurations might be frustrating. Anyway if you want all your data in hiera, also the application specific settings, nodody prevents you from placing in hiera a key with this data and placing it in a variable that you use to feed the options_hash parameter. Really, I have seen and tested different approaches on how to organise things with Puppet: all the data on Hiera, no Hieraat all; class groupings in an ENC, Hiera, a site module; nodes classification using nodes' inheritance, a nodeless setup, an ENC; file managed or settings managed configurations.. I don't have any real reason to say that one approach is better than another, and actually my intention is to find ways (modules) that can adapt to different usage patterns. Hope my clarification exposes better the rationale behind the second part. Yes, I understand your approach... but I don't think it's going very far (given the current trend): Puppet has chosen the multi-parameter route because they want to be able to plug in different data feeds. Neither a config_hash nor a template are really feed-able. Well, if this is the path I think it has to be walked with care, at least for scalability issues: - If you use augeas, file_line or similar to manage per-setting configurations you risk to multiply the number of resources for each node. - If you add many parameters to your classes, besides the extra time needed for the data binding lookups, you have higher module's maintenance costs I'm not saying this should not be done at all, but at the same time that should not be the unique way. Choice, IMHO, reusability is all about freedom of choice. Users' choice, not authors. Al As a side note, a problem with forking the model: when people do fancy stuff (like the truth_enforcer - I loved the idea BTW - or using special libraries as requirements) regular users often can't profit from their modules. Too much effort lost from both sides!! I think PuppetLabs (they have the steering wheel) is more-or-less clear with how they envision the ideal module. As long as we stick to that, our modules will live longer and be used more often. This should not prevent us from thinking of alternatives, obviously. BR/Pablo -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from
Re: [Puppet Users] Re: Rethinking Puppet modules
On Monday, June 23, 2014 10:52:48 AM UTC-5, Alessandro Franceschi wrote: On Friday, June 20, 2014 8:01:49 PM UTC+2, Trevor Vaughan wrote: +1 to everything that John has said. While many Forge modules are not useful to me out of the box, I have had good luck modifying them to meet my needs at any given time. Well, in my very personal opinion, if someone takes a module of mine and has to modify it in order to make it meet his needs, I've lost. We have a deep difference of opinion there. The way I see it, if someone uses a module of mine to serve his needs, with or without modification, then he and I have both won. And if he gives me his modifications (on suitable terms) then I have won twice. In a sense, I can be a bigger winner if users do modify my module than if they don't. I think this may be near the core of our more general difference of opinion about what it means for a module to be 'reusable'. John -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/1e63dc6d-4523-40c3-8e12-1aae136d27a8%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Re: Rethinking Puppet modules
Hi guys, On 06/24/2014 03:48 PM, jcbollinger wrote: I understand the point, but at the same time I find it harmful also to have hundreds of modules, that are published on the forge but work only for a specific case and are practically useless for others. I disagree. It is not at all a problem -- in itself -- to have hundreds of narrowly-scoped modules on the forge. More modules means more chances to find something that serves my needs, or something I can learn from. The concern is presumably about how easy or hard it is to find 'the' right module for me, but I don't think the solution is to try to ensure that all modules published there meet specific, high API and functionality criteria. Rather, the solution is the same as ever in an open-source community: documentation, reputation (both of the code and of the author), popularity (mostly of the code), and support and recommendations from the community, especially (but not exclusively) from thought leaders. I daresay, for example, that many people choose /your/ modules in large part just because they are yours. You and your modules have strong reputations, which are reinforced when users find that your modules work well for them (or undermined in the unlikely event that they find otherwise). Perhaps the Forge platform could be enhanced to make those tools easier to apply in its context, but at least some of them are already reasonably accessible. I have followed this thread with great interest, and although I didn't take the time to read the blog posts, I have to say that I'm pretty much 100% with Alessandro on this one. My own dives into the Forge have been short-lived so far, but the current experience is detrimental, I feel. The multitude of solutions and the fact that there may actually be two or more modules that each suit *some* specific needs of mine, with no visible impetus towards merging similar modules into a one fits all solution makes me despair early. It is my fear that many potential users are repelled by the same experience. It would be my hope that each problem that modules solve could culminate in one module that strikes just the right balance between being generic enough to serve (most) everyone without being overladen with features that almost nobody needs. (Such go-to modules would likely be taken under the umbrella of supported modules.) That is not to say that there should be no ecosystem, we just cannot have too many dead leaves and branches scattered everywhere. Those should be recycled into better core modules by some means. If I read him correctly, Alessandro is currently questing for a way to make just this unification process possible, although that's possibly just my interpretation. (But a common set of practices *will* make the merging of functionality much easier.) I support this. Best, Felix -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/53A98916.9050108%40alumni.tu-berlin.de. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Re: Rethinking Puppet modules
Hi, On 23/06/14 19:15, Alessandro Franceschi wrote: As a side note I'm a bit perplex about the proliferation of parameters in classes or defines in order to match single application specific configuration entries. When I look at this, for example: https://github.com/puppetlabs/puppetlabs-apache/blob/master/manifests/vhost.pp my backbone freezes. A huge amount of define parameters, that map Apache directives, and not even a parameter that allows me to provide a custom template. I agree... it sucks! But I think it sucks more to the person that wrote the module than to those using it. When you see a declaration of a vhost, written in puppet language, it's clear and self-explanatory. But if you pass a template instead, the clarity is lost: you need to look for it elsewhere, it's much more complicated to understand a template than a puppet hash, and you need to seek each variable to see where its value is coming from. The specific problem with vhosts (and other of the kind) is a problem with the language itself: Hiera does not search parameters of defined types, and you need to find your way around with a class that does a simple create_resources to create those resources of that defined type (and probably set defaults with parameters). It could be worse, though :) Also, if you want a pure data driven setup, where you place on hiera hashes of data with all the stuff you need, you can still have this using a template and a single parameters ( such as the options_hash ) I have mixed feelings about this: if it's not a parameter, you don't have automatic hiera lookups, and additionally you have to properly document the structure of the hash. Parameters are almost self-explanatory... but I agree that there are some modules where everything is a parameter, and it's difficult to handle. But I think I would rather have that than doing a template do all the work and have to parameters (template and options_hash) Hope my clarification exposes better the rationale behind the second part. Yes, I understand your approach... but I don't think it's going very far (given the current trend): Puppet has chosen the multi-parameter route because they want to be able to plug in different data feeds. Neither a config_hash nor a template are really feed-able. As a side note, a problem with forking the model: when people do fancy stuff (like the truth_enforcer - I loved the idea BTW - or using special libraries as requirements) regular users often can't profit from their modules. Too much effort lost from both sides!! I think PuppetLabs (they have the steering wheel) is more-or-less clear with how they envision the ideal module. As long as we stick to that, our modules will live longer and be used more often. This should not prevent us from thinking of alternatives, obviously. BR/Pablo -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/53A98ACA.8090800%40cscs.ch. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Re: Rethinking Puppet modules
On Friday, June 20, 2014 8:01:49 PM UTC+2, Trevor Vaughan wrote: +1 to everything that John has said. While many Forge modules are not useful to me out of the box, I have had good luck modifying them to meet my needs at any given time. Well, in my very personal opinion, if someone takes a module of mine and has to modify it in order to make it meet his needs, I've lost. I think, as a general rule, that every Puppet setup should have at least 2 separated directories in the modulepath. A shared, common, one, with public modules downloaded from the Forge or other sources, and not modified. A site, local, one, with local modules and eventually the public modules that have been downloaded and changed (the less of these, the better) for local needs. I've also been able to learn new ways of doing things (and not doing things) due to the willingness of people to share. I do think that some of the recent proposals around allowing multiple modules that cover the same material to exist on the same system are a good thing and could help with reusability in the future. In this case, you would get chunks of reusability that work together but that happily coexist with other isolated chunks. Are you referring to these? https://groups.google.com/forum/#!topic/puppet-dev/em-WGLCrdIw https://groups.google.com/forum/#!topic/puppet-dev/dkdXaUt8yDg It's good to have these types of discussions though. Knowing different users' ideas of perfection can encourage different automated methods of attempting to attain perfection in the future (puppet-lint for example). Indeed, and actually this was the reasoning behind this post, as sometimes I wonder how some things I consider obvious or necessary really matter and are considered useful by others. I generally try to figure out solutions following the current status of the Puppet language (I consider myself an end user) also considering the inevitable lapse of time that occurs from a feature development to its usage at large. Still, proposals as the ones expressed in the linked threads are definitively worth attention, thanks Al On Fri, Jun 20, 2014 at 11:04 AM, jcbollinger john.bo...@stjude.org javascript: wrote: On Thursday, June 19, 2014 11:09:22 AM UTC-5, Alessandro Franceschi wrote: Hi all, I've written a blog post with my opinions on the current modules ecosystem: http://www.example42.com/2014/05/31/rethinking-modules-part-1/ The post talks about : What are the reusability features a module should have, imho The distinction between application (component) modules and higher abstraction modules What are the challenges and reusability options for higher abstraction modules It also raises a pair of points of the challenges that, according to me, we should face in our future modules: Patterns to extend reusability of higher abstraction layer modules Standardisation in the component application modules I'm truly interested in hearing others' opinions on these topics, in order to understand if such points are important only for me or they reflect common concerns. I have never accepted the premise that reusable as applied to Puppet modules has to mean usable in any context, to achieve any end related to the module's area of application, via any conceivable style, which is pretty much how I read your reusability criteria for component modules. Certainly a module that satisfies those criteria can serve pretty much any purpose within its domain, but what matters to me at any given time is whether a module can serve *my* purpose, whatever that happens to be. If some third-party modules are (re)usable for me but not for (say) Felix, and vise versa, then that doesn't change the fact that both our jobs are made easier by not having to write everything from scratch. If that means I need to study a module a bit before I decide whether to use it, then fine -- I ought to do that anyway. Moreover, I think it is harmful to the module ecosystem to promote a position that only Cadillac modules are reusable, because that sets the bar very high for potential contributors. I don't think it helps anyone to tell people that their contributions are unwelcome (which branding them not reusable does). Most people who write modules do so for their own purposes first and foremost, and if the community is unwelcoming to them then they will just keep their work to themselves. I agree, however, that there is a useful distinction between component modules and higher-level modules, and also that some higher-level modules are conceivably reusable. I furthermore agree that the criteria you set out all tend to make such a higher-level module either applicable to more use cases or easier to integrate. On the other hand, I tend to think that higher-level modules will less frequently be used together on the same nodes than are component modules, so that integration
Re: [Puppet Users] Re: Rethinking Puppet modules
Hi, I think you (example42) hit the nail in the head in the rethinking modules part 1 with one tiny exception... but that error was multiplied by x100 in the second part of your dissertation. The exception I am talking about is the desirability to pass a template as a parameter: I agree you should be able to pass a different template, but I think that should be the exception for extreme cases. The idea of the parameters is precisely to avoid passing a file!! In my ideal world, where I manage several heterogeneous systems, I want to be able to glance at one file and see how one node or cluster is configured: that would be the hiera data. Having a complex hierarchy with three or four files to check is already a burden... but it's quite acceptable for the sake of avoiding duplicities. But if I have to check every different template that is passed to the modules to understand what is going on, this capability is completely lost. I am an advocate of the current trend: make good modules that serve everyone (with a restricted purpose), and improve them so that it really serves everyone and they don't to fork (contribute!!). Then higher abstraction modules with wider purpose, configurable so that it serves many people too, and there you go. So, exactly what you described in the part 1... but not in the part 2, sorry. My two cents! BR/Pablo -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/53A85A6F.6020305%40cscs.ch. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Re: Rethinking Puppet modules
On Monday, June 23, 2014 6:49:00 PM UTC+2, pablo.f...@cscs.ch wrote: Hi, I think you (example42) hit the nail in the head in the rethinking modules part 1 with one tiny exception... but that error was multiplied by x100 in the second part of your dissertation. The exception I am talking about is the desirability to pass a template as a parameter: I agree you should be able to pass a different template, but I think that should be the exception for extreme cases. The idea of the parameters is precisely to avoid passing a file!! In my ideal world, where I manage several heterogeneous systems, I want to be able to glance at one file and see how one node or cluster is configured: that would be the hiera data. Having a complex hierarchy with three or four files to check is already a burden... but it's quite acceptable for the sake of avoiding duplicities. But if I have to check every different template that is passed to the modules to understand what is going on, this capability is completely lost. You might be misunderstanding the point about the possibility to pass a template as a reusability option. I'm not saying or advocating the idea of having a template for each node, I just think it's a needed reusability feature. If a class or a define exposes parameters to allow the modification of different variables in a template that's perfectly ok for me, that's the intended usage of templates. As a side note I'm a bit perplex about the proliferation of parameters in classes or defines in order to match single application specific configuration entries. When I look at this, for example: https://github.com/puppetlabs/puppetlabs-apache/blob/master/manifests/vhost.pp my backbone freezes. A huge amount of define parameters, that map Apache directives, and not even a parameter that allows me to provide a custom template. So that, when I want to create a virtualhost file in the way I want, I have to study and accomodate all the parameters to obtain what I need, when I could just pass a custom template and use a few of the core parameters (Virtualhost name, document root and so on) to populate it dynamically. Also, if you want a pure data driven setup, where you place on hiera hashes of data with all the stuff you need, you can still have this using a template and a single parameters ( such as the options_hash here: https://github.com/example42/puppet-tp/blob/master/manifests/conf.pp#L19 ) here you can place any application specific setting without the need to add parameters to the define (in your template, of course, you have to use and manage this hash). The intended usage of defines like tp::conf which rely entirely on external templates or files, is inside higher abstraction modules, where you can expose the parameters you want and which provide a specific template that may use the exposed parameters, keeping static what you don't need to modify in your nodes. I am an advocate of the current trend: make good modules that serve everyone (with a restricted purpose), and improve them so that it really serves everyone and they don't to fork (contribute!!). Then higher abstraction modules with wider purpose, configurable so that it serves many people too, and there you go. So, exactly what you described in the part 1... but not in the part 2, sorry. Hope my clarification exposes better the rationale behind the second part. Thank you Al My two cents! BR/Pablo -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/ea615159-033d-4edf-aafd-47545a4b5f24%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Re: Rethinking Puppet modules
+1 to everything that John has said. While many Forge modules are not useful to me out of the box, I have had good luck modifying them to meet my needs at any given time. I've also been able to learn new ways of doing things (and not doing things) due to the willingness of people to share. I do think that some of the recent proposals around allowing multiple modules that cover the same material to exist on the same system are a good thing and could help with reusability in the future. In this case, you would get chunks of reusability that work together but that happily coexist with other isolated chunks. It's good to have these types of discussions though. Knowing different users' ideas of perfection can encourage different automated methods of attempting to attain perfection in the future (puppet-lint for example). Thanks, Trevor On Fri, Jun 20, 2014 at 11:04 AM, jcbollinger john.bollin...@stjude.org wrote: On Thursday, June 19, 2014 11:09:22 AM UTC-5, Alessandro Franceschi wrote: Hi all, I've written a blog post with my opinions on the current modules ecosystem: http://www.example42.com/2014/05/31/rethinking-modules-part-1/ The post talks about : What are the reusability features a module should have, imho The distinction between application (component) modules and higher abstraction modules What are the challenges and reusability options for higher abstraction modules It also raises a pair of points of the challenges that, according to me, we should face in our future modules: Patterns to extend reusability of higher abstraction layer modules Standardisation in the component application modules I'm truly interested in hearing others' opinions on these topics, in order to understand if such points are important only for me or they reflect common concerns. I have never accepted the premise that reusable as applied to Puppet modules has to mean usable in any context, to achieve any end related to the module's area of application, via any conceivable style, which is pretty much how I read your reusability criteria for component modules. Certainly a module that satisfies those criteria can serve pretty much any purpose within its domain, but what matters to me at any given time is whether a module can serve *my* purpose, whatever that happens to be. If some third-party modules are (re)usable for me but not for (say) Felix, and vise versa, then that doesn't change the fact that both our jobs are made easier by not having to write everything from scratch. If that means I need to study a module a bit before I decide whether to use it, then fine -- I ought to do that anyway. Moreover, I think it is harmful to the module ecosystem to promote a position that only Cadillac modules are reusable, because that sets the bar very high for potential contributors. I don't think it helps anyone to tell people that their contributions are unwelcome (which branding them not reusable does). Most people who write modules do so for their own purposes first and foremost, and if the community is unwelcoming to them then they will just keep their work to themselves. I agree, however, that there is a useful distinction between component modules and higher-level modules, and also that some higher-level modules are conceivably reusable. I furthermore agree that the criteria you set out all tend to make such a higher-level module either applicable to more use cases or easier to integrate. On the other hand, I tend to think that higher-level modules will less frequently be used together on the same nodes than are component modules, so that integration considerations are less of an issue. I also tend to think that reusable higher-level modules will be fewer than component modules, so that standardization is less of a consideration for them. Of course, I think I ascribe less importance overall to standardization (of Puppet modules) than you do. I have recently decided, too, that more of the responsibility for module reusability has been placed on module authors than is needful or useful. Puppet itself could and should provide more support than it does. In particular, I would like to see more flexible data binding (e.g. a mechanism for data aliases and/or indirect data names, a mechanism for slicing data, improved DSL-based data injection, etc.), and more flexible class / module resolution (e.g. aliases / indirection). These sorts of features would improve *all* modules' reusability by opening more flexible approaches to integrating modules into a site's manifest set. John -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/e30a76b9-3f1a-4408-8cc4-d4c2a2207874%40googlegroups.com
