[Puppet - Bug #17156] Regression: Can't restrict the plugins loaded by puppet doc with --modulepath in 3.0.1
Issue #17156 has been updated by Josh Cooper. I'm digging into this issue as it relates to #7316. I'm actually having trouble reproducing the problem, see comments below: Nick Fagerlund wrote: In 3.0.0, puppet doc would ONLY load plugins from the modulepath; the contents of the agent’s libdir had no effect, and changing the modulepath swapped out the entire set of plugins available. The doc application uses an InstanceLoader when generating reference documentation. The InstanceLoader delegates to Autoload(er)s, one per reference type (function, report, etc). In 3.0.0, the Autoloader's search path always includes the current environment's modulepath and the libdir. However, in 3.0.0, when running in `master` run_mode, libdir will refer to the one in /etc/puppet/var/lib, even when not running as root. For example, I see the autoloader using my modulepaths and system libdir: pre /tmp/modules/user/cloud_provisioner/lib, /tmp/modules/user/mod1/lib, /tmp/modules/user/mod2/lib, /var/lib/puppet/lib, /pre In 3.0.1, puppet doc loads plugins from both the modulepath AND the libdir. So if puppet agent has ever run, puppet doc will keep loading plugins until you delete the libdir. In 3.0.1, the run_mode no longer affects which vardir (and libdir) is used. Instead, it's solely based on whether you're running as root or not. As a result, I see it using my user's libdir: pre /tmp/modules/user/cloud_provisioner/lib, /tmp/modules/user/mod1/lib, /tmp/modules/user/mod2/lib, /Users/josh/.puppet/var/lib, /pre So it's not that it started using the libdir, just that it started using a different one than in 3.0.0. So I've thought about this a little more. Principles: * Puppet doc, in its reference mode, should only document components that are **installed.** That makes sense. So in conclusion: * Having puppet doc read plugins from the **agent's** libdir is wrong. * Having it read plugins from the **master's** libdir is right. In 3.0.1 and above, there is not distinction between master and agent libdir. It's solely based on whether you're running as root or not. * Obviously the results will be stupid if the agent and the master share a libdir, just like everything else gets stupid. * And what it's probably actually doing is using the libdir from the `[user]` config block, which is approximately as wrong as using the agent's libdir. So given that... I suspect it makes sense to switch puppet doc to master run mode. That way, the theoretical users who're generating docs for their custom types and functions will get the correct behavior pretty much no matter what. The doc application currently runs in master run_mode. ... Does that sound right? Also I may be overthinking this, as we're probably the only ones who use reference mode. :/ If you only want to generate a reference for **core** types, etc, ignoring installed modules and pluginsync'ed libdir, you could do: pre $ puppet doc --reference [..] --libdir /dev/null --modulepath /dev/null /pre If you want to generate a reference for **installed** types, excluding libdir: pre $ puppet doc --reference [..] --libdir /dev/null /pre If you want to generate a reference for **installed** types, **including** those in the libdir (on the master): pre $ puppet doc --reference [..] /pre All of this assumes you are running `puppet doc` as the same user as you ran `puppet module install`, which I think is what people would expect. Using 3.0.2, and a puppet.conf of: pre [master] modulepath=/tmp/modules/user /pre If I install a module: pre $ puppet module install puppetlabs-stdlib Notice: Preparing to install into /tmp/modules/user ... /pre And then I can generate documentation for it, including the module I just installed: pre $ puppet doc --reference function ... abs bool2num .. /pre Does that make sense? Bug #17156: Regression: Can't restrict the plugins loaded by puppet doc with --modulepath in 3.0.1 https://projects.puppetlabs.com/issues/17156#change-78401 Author: Nick Fagerlund Status: Accepted Priority: Normal Assignee: Category: settings Target version: 3.x Affected Puppet version: 3.0.1 Keywords: regression settings Branch: The docs team uses `puppet doc --reference` to generate certain docs for the site, and we have to use `--modulepath /tmp/nothing` or something to exclude non-core resource types and functions from our personal modules. This worked through 3.0.0 and broke in 3.0.1. The changed behavior: * In 3.0.0, puppet doc would ONLY load plugins from the modulepath; the contents of the agent's libdir had no effect, and changing the modulepath swapped out the entire set of plugins available. * In 3.0.1, puppet doc loads plugins from both the modulepath AND the libdir. So if puppet agent has ever run, puppet doc will keep loading plugins until you delete the libdir. Git bisect tracks the change to
[Puppet - Bug #17156] Regression: Can't restrict the plugins loaded by puppet doc with --modulepath in 3.0.1
Issue #17156 has been updated by Andrew Parker. Nick Fagerlund wrote: This also shows me the way to a new workaround, which is to blank out the modulepath AND the libdir when running puppet doc. That dials down the urgency here, and it's possible (but not likely?) that we actually WANT the new behavior. Not closing the bug though, because this is an unintentional change and we shouldn't close it until we have made a design decision about how puppet doc should work. Nick, after our talk about this, I think we came to the conclusion that what it is doing is what it should be doing. Can you confirm that, and if I'm remembering correctly, close this? Bug #17156: Regression: Can't restrict the plugins loaded by puppet doc with --modulepath in 3.0.1 https://projects.puppetlabs.com/issues/17156#change-74992 Author: Nick Fagerlund Status: Accepted Priority: Normal Assignee: Category: settings Target version: 3.x Affected Puppet version: 3.0.1 Keywords: regression settings Branch: The docs team uses `puppet doc --reference` to generate certain docs for the site, and we have to use `--modulepath /tmp/nothing` or something to exclude non-core resource types and functions from our personal modules. This worked through 3.0.0 and broke in 3.0.1. The changed behavior: * In 3.0.0, puppet doc would ONLY load plugins from the modulepath; the contents of the agent's libdir had no effect, and changing the modulepath swapped out the entire set of plugins available. * In 3.0.1, puppet doc loads plugins from both the modulepath AND the libdir. So if puppet agent has ever run, puppet doc will keep loading plugins until you delete the libdir. Git bisect tracks the change to commit:144bb76a7e825a221fd1cb5a20728c8cf5c03264 -- (#16637) Fix default confdir and vardir with puppet master as normal user. The command to test this with is `puppet doc --reference function --modulepath /tmp/nothing` -- make sure you have something like puppetlabs-stdlib in your modulepath when testing, so that the function reference gets nice and messy. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To post to this group, send email to puppet-bugs@googlegroups.com. To unsubscribe from this group, send email to puppet-bugs+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-bugs?hl=en.
[Puppet - Bug #17156] Regression: Can't restrict the plugins loaded by puppet doc with --modulepath in 3.0.1
Issue #17156 has been updated by Nick Fagerlund. So I've thought about this a little more. Principles: * Puppet doc, in its reference mode, should only document components that are **installed.** * Plugins cached by puppet agent during a run aren't installed; they're a temporary, volatile reflection of what's installed on the master. * But plugins in every environment on the master are all installed! And our solution for reading everything in every environment was to copy it all to the master's libdir. * Dunno what to say about puppet apply except that it sort of cares about environments, but in practice it really doesn't. So in conclusion: * Having puppet doc read plugins from the **agent's** libdir is wrong. * Having it read plugins from the **master's** libdir is right. * Obviously the results will be stupid if the agent and the master share a libdir, just like everything else gets stupid. * And what it's probably actually doing is using the libdir from the `[user]` config block, which is approximately as wrong as using the agent's libdir. So given that... I suspect it makes sense to switch puppet doc to master run mode. That way, the theoretical users who're generating docs for their custom types and functions will get the correct behavior pretty much no matter what. * If they're doing this on the puppet master, it'll document all plugins from all environments, as expected. * If they're doing it on a puppet apply standalone node, most users are probably putting their settings in `[main]` instead of `[user]` (that's a hunch) and leaving `[master]` blank, so since master has no secret sauce anymore, they'll get correct behavior too. * And if I'm right in my hunch, behavior will be correct for the module docs mode as well as the reference mode. If I'm wrong, there are some rare users who will get a surprise, but it won't be a huge one. * If they're doing it on an arbitrary agent, they're doing it wrong. * Docs team's use case remains idiosyncratic, and that's fine, we'll just blank out the libdir on the command line. Does that sound right? Also I may be overthinking this, as we're probably the only ones who use reference mode. :/ Bug #17156: Regression: Can't restrict the plugins loaded by puppet doc with --modulepath in 3.0.1 https://projects.puppetlabs.com/issues/17156#change-74996 Author: Nick Fagerlund Status: Accepted Priority: Normal Assignee: Category: settings Target version: 3.x Affected Puppet version: 3.0.1 Keywords: regression settings Branch: The docs team uses `puppet doc --reference` to generate certain docs for the site, and we have to use `--modulepath /tmp/nothing` or something to exclude non-core resource types and functions from our personal modules. This worked through 3.0.0 and broke in 3.0.1. The changed behavior: * In 3.0.0, puppet doc would ONLY load plugins from the modulepath; the contents of the agent's libdir had no effect, and changing the modulepath swapped out the entire set of plugins available. * In 3.0.1, puppet doc loads plugins from both the modulepath AND the libdir. So if puppet agent has ever run, puppet doc will keep loading plugins until you delete the libdir. Git bisect tracks the change to commit:144bb76a7e825a221fd1cb5a20728c8cf5c03264 -- (#16637) Fix default confdir and vardir with puppet master as normal user. The command to test this with is `puppet doc --reference function --modulepath /tmp/nothing` -- make sure you have something like puppetlabs-stdlib in your modulepath when testing, so that the function reference gets nice and messy. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To post to this group, send email to puppet-bugs@googlegroups.com. To unsubscribe from this group, send email to puppet-bugs+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-bugs?hl=en.
[Puppet - Bug #17156] Regression: Can't restrict the plugins loaded by puppet doc with --modulepath in 3.0.1
Issue #17156 has been updated by Andrew Parker. Strangely enough, `puppet doc` should already be running in the master mode: https://github.com/puppetlabs/puppet/blob/master/lib/puppet/application/doc.rb#L4. However, on the line right above that it specifies that it shouldn't parse configs. Down near the bottom of the file it tries to do some fancy footwork to parse things in its own special way. I have no idea what this all achieves in the end. What you are saying makes perfect sense, but I don't fully understand what it is doing right now so I'm not clear how far off it currently is. It does look like it never looks at the libdir, but instead only really cares about the modulepath. Bug #17156: Regression: Can't restrict the plugins loaded by puppet doc with --modulepath in 3.0.1 https://projects.puppetlabs.com/issues/17156#change-75000 Author: Nick Fagerlund Status: Accepted Priority: Normal Assignee: Category: settings Target version: 3.x Affected Puppet version: 3.0.1 Keywords: regression settings Branch: The docs team uses `puppet doc --reference` to generate certain docs for the site, and we have to use `--modulepath /tmp/nothing` or something to exclude non-core resource types and functions from our personal modules. This worked through 3.0.0 and broke in 3.0.1. The changed behavior: * In 3.0.0, puppet doc would ONLY load plugins from the modulepath; the contents of the agent's libdir had no effect, and changing the modulepath swapped out the entire set of plugins available. * In 3.0.1, puppet doc loads plugins from both the modulepath AND the libdir. So if puppet agent has ever run, puppet doc will keep loading plugins until you delete the libdir. Git bisect tracks the change to commit:144bb76a7e825a221fd1cb5a20728c8cf5c03264 -- (#16637) Fix default confdir and vardir with puppet master as normal user. The command to test this with is `puppet doc --reference function --modulepath /tmp/nothing` -- make sure you have something like puppetlabs-stdlib in your modulepath when testing, so that the function reference gets nice and messy. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To post to this group, send email to puppet-bugs@googlegroups.com. To unsubscribe from this group, send email to puppet-bugs+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-bugs?hl=en.
[Puppet - Bug #17156] Regression: Can't restrict the plugins loaded by puppet doc with --modulepath in 3.0.1
Issue #17156 has been updated by Nick Fagerlund. Subject changed from Regression: Can't pass --modulepath (or any other config setting flag) to puppet doc in 3.0.1 to Regression: Can't restrict the plugins loaded by puppet doc with --modulepath in 3.0.1 Description updated No, wait, I was way too hasty about deciding what caused the problem here. It turns out to be a libdir issue! Changing description accordingly. And now that I think about it, it's possible that puppet doc was always TRYING to do the current behavior, but was botching it and looking in the wrong libdir. Bug #17156: Regression: Can't restrict the plugins loaded by puppet doc with --modulepath in 3.0.1 https://projects.puppetlabs.com/issues/17156#change-74649 Author: Nick Fagerlund Status: Accepted Priority: Normal Assignee: Category: settings Target version: 3.x Affected Puppet version: 3.0.1 Keywords: regression settings Branch: The docs team uses `puppet doc --reference` to generate certain docs for the site, and we have to use `--modulepath /tmp/nothing` or something to exclude non-core resource types and functions from our personal modules. This worked through 3.0.0 and broke in 3.0.1. The changed behavior: * In 3.0.0, puppet doc would ONLY load plugins from the modulepath; the contents of the agent's libdir had no effect, and changing the modulepath swapped out the entire set of plugins available. * In 3.0.1, puppet doc loads plugins from both the modulepath AND the libdir. So if puppet agent has ever run, puppet doc will keep loading plugins until you delete the libdir. Git bisect tracks the change to commit:144bb76a7e825a221fd1cb5a20728c8cf5c03264 -- (#16637) Fix default confdir and vardir with puppet master as normal user. The command to test this with is `puppet doc --reference function --modulepath /tmp/nothing` -- make sure you have something like puppetlabs-stdlib in your modulepath when testing, so that the function reference gets nice and messy. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To post to this group, send email to puppet-bugs@googlegroups.com. To unsubscribe from this group, send email to puppet-bugs+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-bugs?hl=en.
[Puppet - Bug #17156] Regression: Can't restrict the plugins loaded by puppet doc with --modulepath in 3.0.1
Issue #17156 has been updated by Nick Fagerlund. This also shows me the way to a new workaround, which is to blank out the modulepath AND the libdir when running puppet doc. That dials down the urgency here, and it's possible (but not likely?) that we actually WANT the new behavior. Not closing the bug though, because this is an unintentional change and we shouldn't close it until we have made a design decision about how puppet doc should work. Bug #17156: Regression: Can't restrict the plugins loaded by puppet doc with --modulepath in 3.0.1 https://projects.puppetlabs.com/issues/17156#change-74650 Author: Nick Fagerlund Status: Accepted Priority: Normal Assignee: Category: settings Target version: 3.x Affected Puppet version: 3.0.1 Keywords: regression settings Branch: The docs team uses `puppet doc --reference` to generate certain docs for the site, and we have to use `--modulepath /tmp/nothing` or something to exclude non-core resource types and functions from our personal modules. This worked through 3.0.0 and broke in 3.0.1. The changed behavior: * In 3.0.0, puppet doc would ONLY load plugins from the modulepath; the contents of the agent's libdir had no effect, and changing the modulepath swapped out the entire set of plugins available. * In 3.0.1, puppet doc loads plugins from both the modulepath AND the libdir. So if puppet agent has ever run, puppet doc will keep loading plugins until you delete the libdir. Git bisect tracks the change to commit:144bb76a7e825a221fd1cb5a20728c8cf5c03264 -- (#16637) Fix default confdir and vardir with puppet master as normal user. The command to test this with is `puppet doc --reference function --modulepath /tmp/nothing` -- make sure you have something like puppetlabs-stdlib in your modulepath when testing, so that the function reference gets nice and messy. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To post to this group, send email to puppet-bugs@googlegroups.com. To unsubscribe from this group, send email to puppet-bugs+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-bugs?hl=en.
