Re: [Puppet-dev] Ruby environment variable handling in Puppet Server

[Clayton O'Neill]

 

I'm not sure that we have a use case for the server, but as an aside, we do 
have 
a use case on the agent side, if that is ever a concern:

We use the ability to set environment variables when doing OpenStack upgrades.  
OpenStack has the concept of public (publicURL) and private (internalURL) 
endpoints.  Normally during our Puppet runs it uses the public endpoints 
for all API calls and provisioning.  However, when we're doing upgrades, we 
sometimes have the external load balancer disabled so that clients can not 
make changes to a database we may revert.  In this case, we set the 
OS_ENDPOINT_TYPE environment variable to 'internalURL' in our upgrade 
scripts.

I don't know if this will ever be an issue for the agent, but in this situation 
a config based option would be significantly less useful.

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/2c889e90-8514-4c8c-836b-93633a2e904b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Puppet environments implementation question - get_conf() location and result caching

[Bostjan Skufca]

On Thursday, 16 April 2015 14:01:29 UTC+2, henrik lindberg wrote:
>
> On 2015-16-04 4:30, Bostjan Skufca wrote: 
> > Just recently I was looking at the environments part of puppet 
> > implementation, and I have stumbled upon this curious gimmick. 
> > 
> > There is Puppet::Environments class, which is a class for general 
> > Environments management (searching, loaders, caching, etc). 
> > Then there is Puppet::Node::Environment which is instantiated for every 
> > environent found. Logical up to this point. 
> > 
> > But, contrary to all OOP notions, get_conf() function is not a method of 
> > Puppet::Node::Environment class. 
> > Rather, it is a lookup function in Puppet::Environments class which 
> > takes environment name as an argument and fetches 
> > Puppet::Settings::EnvironmentConf instance. 
> > 
> > Puppet::Environments has two function for environment and environment 
> > data retrieval: 
> > - get() fetches Puppet::Node::Environment instance 
> > - get_conf() fetches Puppet::Settings::EnvironmentConf instance 
> > 
> > The curious thing is: 
> > - get() method operates with cache 
> > - get_conf() does not include any caching 
> > 
> > I tested this on a catalog of 1633 resources, and: 
> > - get() was called around 220 times 
> > - get_conf() was called around 25.000 times 
> > 
> > These are the comments above get_conf() in environments.rb: 
> >  # This implementation evicts the cache, and always gets the current 
> >  # configuration of the environment 
> >  # 
> >  # TODO: While this is wasteful since it 
> >  # needs to go on a search for the conf, it is too disruptive to 
> > optimize 
> >  # this. 
> > 
> > My questions are: 
> > - why does get_conf() method belong to Environments instead of 
> > Node::Environment class? 
> > - why are get_conf() results not cached? Why is it "too disruptive"? 
> > 
>
> It should be fine to cache the env conf in an instance of an 
> environment, and it should be possible to obtain the config given 
> an instantiated environment. This is safe since the conf will then be 
> evicted when the environment is evicted (and while the environment is 
> cached it will not reload or change).


It would be more logical that way too, for the user. If there is 
"environment timeout", user would expect that all parts of environment (env 
configuration, code, even manifests) are cached. At least that was my 
understanding until I actually looked at the code.

Shall I create a Jira ticket and pull request with working code change?

 

> I believe that the large number of calls 
> to get the conf could almost exclusively be for an already loaded 
> environment (I should be measured though). 
>

Yes, this is true. All the calls are for the environment where the 
node/agent resides. I tested this with single node only, though.

 

> All other calls would be for information purposes and those cannot be 
> cached unless the environment.conf file is watched (which is difficult 
> since it may not exist, come into existence, be removed etc.). For those 
> calls, the cost of checking if the file is up to date is almost as 
> expensive as loading the conf. 
>

"All other calls" - do you mean calls for other environments?
If so, as said above, I did not notice this on my setup (could not). But I 
believe that if env caching follows the same pattern for all environments, 
it should not be a problem.


Since the 3.x implementation is very complex due to the dynamic 
> environments, I suggest that this should be fixed for 4.x. 
>

This is probably for the best, true. If one wants to refrain from using 
dynamic environments and wishes to use 3.7.x, a separate patch may be 
provided.


b.

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/57e71df8-0a83-4cd7-a851-6f2fc2e0954c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet-dev] Ruby environment variable handling in Puppet Server

[Jeremy Barlow]

Thanks for the responses on this thread so far and some of the 
corresponding discussion that has been spawned in the related JIRA ticket 
- https://tickets.puppetlabs.com/browse/SERVER-297.

On Tuesday, April 14, 2015 at 9:09:40 AM UTC-7, Nan Liu wrote:
>
> On Tuesday, April 14, 2015 at 4:35:29 AM UTC-7, Ken Barber wrote:
>>
>> > While we are leaning toward a config-file driven approach, we would be 
>> > interested in hearing of any specific use cases you may know of where 
>> this 
>> > may be insufficient.  We would specifically be interested in any use 
>> cases 
>> > which suggest that some affordance in the design should be made to 
>> allow for 
>> > some (or all?) variables seen by Ruby code to be drawn from the actual 
>> shell 
>> > environment, as opposed to just a configuration file. 
>>
>> Might be clutching at straws here, but there might be a case for 
>> something like http_proxy (which is a reasonably common convention) in 
>> a closed environment that requires it, to be just passed through, 
>> versus defining it also in another configuration file again. That kind 
>> of environment var is _sometimes_ set globally to avoid configuring 
>> the proxy config in all the different clients/services that a *nix box 
>> has. I think Net::HTTP honors this environment variable for example, 
>> so this might apply to some functions that make outbound http calls. 
>>
>
> +1, http_proxy and no_proxy not being honored in puppet functions is one 
> of the annoyances I've run into with puppet-server. 
>
> Of course, I'd rather here what the community has to say about this. 
>> Maybe users would prefer to manage this more precisely instead of 
>> globally anyway from a puppetserver/function perspective. 
>
>
> I'm fine explicitly setting environment variable for puppetserver if 
> there's an option to passthrough
>

Most of the discussion I've seen about this so far has centered on the lack 
of an ability for Puppet Server to use a proxy for HTTP/S communications 
when needed.  This has been mostly with respect to the "puppetserver gem" 
command being unable to access gem repositories via a proxy - also covered 
in https://tickets.puppetlabs.com/browse/SERVER-377.  Proxy support is an 
issue that clearly needs to be addressed, both for puppetserver CLI tools 
and the production Puppet Server stack.  See 
https://tickets.puppetlabs.com/browse/SERVER-156 around the production 
Puppet Server stack's current lack of support for using a proxy.

We will certainly need to get to a solution that allows for values for the 
*_PROXY variables to be made available to Ruby code running in a JRuby 
container in Puppet Server.  I think there's also a very reasonable case to 
be made that these specific variables be drawn from the actual shell 
environment, when not overridden by Puppet Server configuration, given that 
they are very commonly used to perform system-wide proxy configuration that 
many tools honor as defaults.

What is not clear to me at this point, though, is whether the *_PROXY 
variable case alone is enough to inform the more general approach that we 
take toward environment variable population into the JRuby containers - and 
specifically whether or not there is a requirement to provide a 
more-general purpose mechanism for choosing arbitrary variables to flow 
through from the shell environment to the JRuby container.

I'd like to hear if anyone has other use cases - ones not related to 
*_PROXY - for flowing environment variables from the shell to the JRuby 
container.  Specifically, I'd like to hear of other use cases that a 
config-file driven approach like the one described in the initial post on 
this thread would not satisfy.

Thanks again!

--- Jeremy

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/a2f499fa-5f39-40eb-ab1c-d523a445391e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Puppet environments implementation question - get_conf() location and result caching

[Henrik Lindberg]

On 2015-16-04 4:30, Bostjan Skufca wrote:

Just recently I was looking at the environments part of puppet
implementation, and I have stumbled upon this curious gimmick.

There is Puppet::Environments class, which is a class for general
Environments management (searching, loaders, caching, etc).
Then there is Puppet::Node::Environment which is instantiated for every
environent found. Logical up to this point.

But, contrary to all OOP notions, get_conf() function is not a method of
Puppet::Node::Environment class.
Rather, it is a lookup function in Puppet::Environments class which
takes environment name as an argument and fetches
Puppet::Settings::EnvironmentConf instance.

Puppet::Environments has two function for environment and environment
data retrieval:
- get() fetches Puppet::Node::Environment instance
- get_conf() fetches Puppet::Settings::EnvironmentConf instance

The curious thing is:
- get() method operates with cache
- get_conf() does not include any caching

I tested this on a catalog of 1633 resources, and:
- get() was called around 220 times
- get_conf() was called around 25.000 times

These are the comments above get_conf() in environments.rb:
 # This implementation evicts the cache, and always gets the current
 # configuration of the environment
 #
 # TODO: While this is wasteful since it
 # needs to go on a search for the conf, it is too disruptive to
optimize
 # this.

My questions are:
- why does get_conf() method belong to Environments instead of
Node::Environment class?
- why are get_conf() results not cached? Why is it "too disruptive"?



The path to the new environments implementation (supporting both 
directory environments, and the now deprecated (and in 4.0 removed) 
dynamic environments) was paved with problems as it affected more or

less the entire code base.

It should be fine to cache the env conf in an instance of an 
environment, and it should be possible to obtain the config given
an instantiated environment. This is safe since the conf will then be 
evicted when the environment is evicted (and while the environment is 
cached it will not reload or change).


I believe that the large number of calls
to get the conf could almost exclusively be for an already loaded 
environment (I should be measured though).
All other calls would be for information purposes and those cannot be 
cached unless the environment.conf file is watched (which is difficult 
since it may not exist, come into existence, be removed etc.). For those 
calls, the cost of checking if the file is up to date is almost as 
expensive as loading the conf.


Since the 3.x implementation is very complex due to the dynamic 
environments, I suggest that this should be fixed for 4.x.


- henrik


Thanks for answers,
b.

--
You received this message because you are subscribed to the Google
Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-dev+unsubscr...@googlegroups.com
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/53b560d5-1c8f-4b00-944c-3156343a175f%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mgo8ac%24hc5%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.