On Sat, Mar 20, 2010 at 8:46 AM, Brice Figureau
<[email protected]> wrote:
> On 20/03/10 14:56, Nigel Kersten wrote:
>> On Sat, Mar 20, 2010 at 3:45 AM, Brice Figureau
>> <[email protected]> wrote:
>>> On 20/03/10 02:51, Nigel Kersten wrote:
>>>> So this seemed worth another thread given the significant progress...
>>>> just about all done by Brice as usual.... :)
>>>>
>>>> With jruby 1.4.0 and jetty-rackup from
>>>> http://github.com/geekq/jetty-rackup and a simple config.ru file that
>>>> looks like:
>>>
>>> I'm using without issue:
>>> jruby 1.5.0.dev (ruby 1.8.7 patchlevel 174) (2010-03-08 1cc36d4)
>>>
>>>> -----------------
>>>> $0 = "puppetmasterd"
>>>> require 'puppet'
>>>>
>>>> ARGV << "--verbose"
>>>> #ARGV << "--debug"
>>>> #ARGV << "--trace"
>>>> # I have a special config file for my servers
>>>> ARGV << "--config" << "/etc/puppet/puppetmasterd.conf"
>>>> ARGV << "--no-daemonize"
>>>> ARGV << "--user" << "puppet"
>>>> ARGV << "--no-ca"
>>>>
>>>> ARGV << "--rack"
>>>> require 'puppet/application/puppetmasterd'
>>>> # we're usually running inside a Rack::Builder.new {} block,
>>>> # therefore we need to call run *here*.
>>>> run Puppet::Application[:puppetmasterd].run
>>>> -----------------
>>>>
>>>> then this little modification to jetty-rackup
>>>> -----------------
>>>> jetty = org.mortbay.jetty.Server.new options[:Port]
>>>> # next 3 lines are new
>>>> jetty.get_connectors.each do |c|
>>>> c.set_header_buffer_size 8192
>>>> end
>>>> -----------------
>>>
>>> I was about to send you this one :-) but you figured out it.
>>>
>>>> and we have something that almost works.
>>>>
>>>> err: Could not retrieve catalog: Uncaught exception fork is unsafe and
>>>> disabled by default on JRuby in method puppetmaster.getconfig
>>>
>>> It's strange I don't get that.
>>>
>>> Maybe we could rewrite Puppet::Util#execute for jruby to use the JVM
>>> execute instead of forking.
>>>
>>>> I've enabled forking.... -J-Djruby.fork.enabled=true and crossed my
>>>> fingers... oh look at all this...
>>>>
>>>> WARNING: fork is highly unlikely to be safe or stable on the JVM. Have fun!
>>>> WARNING: fork is highly unlikely to be safe or stable on the JVM. Have fun!
>>>> WARNING: fork is highly unlikely to be safe or stable on the JVM. Have fun!
>>>>
>>>> and we blow up earlier than we did before... Tried the latest HEAD,
>>>> it was completely broken, so reverted to a few commits earlier, and
>>>> made some progress... but then the actual forking looks to fail.
>>>>
>>>> info: Autoloaded module base
>>>> debug: Puppet::Type::Package::ProviderRpm: Executing '/usr/bin/rpm
>>>> --version'
>>>> Could not autoload rpm: No child processes - No child processes
>>>
>>> OK, in fact looking harder, I have this call to execute, but it fails in
>>> my JRuby version without impacting the compilation itself.
>>>
>>> In fact the issue comes from this code from the rpm provider:
>>> if command('rpm')
>>> confine :true => begin
>>> rpm('--version')
>>> rescue Puppet::ExecutionFailure
>>> false
>>> else
>>> true
>>> end
>>> end
>>>
>>> This code is executed because each type when compiling is asked if it's
>>> a builtin type, which in turn autoload each provider, which in turn
>>> executes such "global" code (ie checking confine).
>>> I don't think it is necessary to autoload the providers (type should be
>>> enough for the master, isn't it?), but I'm afraid it isn't possible.
>>
>> So when I thought about this more, I got more confused. I really
>> thought providers were pretty much ignored by the server. This seems
>> quite undesirable to have any of this come into play at all on the
>> server.
>
> I was surprised to find this too. I really thought we weren't loading
> any provider on the server.
>
>> I take it you're saying that it isn't possible to just autoload the
>> type without the provider?
>
> I don't see why it shouldn't be possible, but looking the type auto
> loader code there is this:
>
> # Define a new type.
> def newtype(name, options = {}, &block)
> ...
> # Now set up autoload any providers that might exist for this type.
> klass.providerloader = Puppet::Util::Autoload.new(klass,
> "puppet/provider/#{klass.name.to_s}"
> )
>
> # We have to load everything so that we can figure out the default type.
> klass.providerloader.loadall()
> ...
> end
>
> I don't think it's necessary to load the provider on the server, but
> this last comment looks strange to me.
>
> I think it is safe to comment those lines in puppet/metatype/manager.rb
> for your experiment.
yeah, no difference with a quick test. I'll do some more tracing later
to work out where this is coming from.
>
>>>
>>> Anyway if you checkout jruby 1cc36d4 and use the default option you get
>>> a nice stack trace but no failure of the compilation :-)
>>
>> ahah ;) I'll give that a try.
>>
>>>
>>>> It looks like we only fork in three places in Puppet for what it's worth.
>>>
>>> Yes, it's:
>>> - when daemonizing which you and I are not doing
>>> - when executing (or generate)
>>> - in puppetrun
>>>
>>> I think execute should be rewritten to use the JVM Runtime.exe or
>>> ProcessBuilder for JRuby, instead of forking. I'll see if I can come
>>> with a patch later in the week-end.
>>
>> I was thinking the same thing, I'll have a poke around too.
>
> Something along of:
>
> require 'java'
>
> include_class 'java.lang.Runtime'
> include_class 'java.io.InputStreamReader'
> include_class 'java.io.BufferedReader'
>
> process = Runtime.runtime.exec(command)
> stream = BufferedReader.new(
> InputStreamReader.new(
> process.input_stream
> )
> )
>
> output = ""
> while line = stream.read_line
> output << "#{line}\n"
> end
>
> process.wait_for
>
> should do it fine
For normal execution :)
Our execute method is a little weirder with all the failonfail,
stdinfile and combine args, some of which some of my providers rely
on.
Let's assume we're in the happy place where JRuby is working perfectly
apart from this issue. How would we maintain this cleanly in the code
base?
> --
> Brice Figureau
> My Blog: http://www.masterzen.fr/
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Developers" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected].
> For more options, visit this group at
> http://groups.google.com/group/puppet-dev?hl=en.
>
>
--
nigel
--
You received this message because you are subscribed to the Google Groups
"Puppet Developers" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/puppet-dev?hl=en.
