Just an update. This is what strace shows at the point where the process
finishes fetching data from the MySQL database:
17169 read(4, "38:31\373\373\0232013-07-15 17:38:32\373\023201"..., 16384)
= 384
17169 read(4, "3:00:07\0074113192\27xavier.ofi.softo"..., 16384) = 16000
17169 read(4, "305566\17some.user\003194\00222\00741"..., 16384) = 384
17169 read(4, "192\0232013-04-02 16:11:44\0232013-04-"..., 16384) = 8534
17169 gettimeofday({1381493430, 91178}, NULL) = 0
17169 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
17169 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
17169 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
17169 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
17169 mmap(NULL, 49348608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0x7f41c7266000
17169 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
17169 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
17169 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
Beyond this point, tons of exactly the same line, CPU at 100% and memory
usage increasing until it fills RAM and swap space, and then the SIGKILL
shown above.
Roger
On Tuesday, October 8, 2013 7:22:20 PM UTC+2, Roger Torrentsgenerós wrote:
>
>
> Hi,
>
> I currently have a MySQL database containing all Puppet storeconfigs. My
> intention is to migrate to PuppetDB on a PostgreSQL server, so the first
> step is to use the 'storeconfigs' face to export all the DB to a file
> PuppetDB can later consume. But the 'puppet storeconfigs export' command
> always ends up being killed, I suspect due to some sort of OOM issue. I
> have tried different environments, different Puppet / Ruby / whatever
> versions, different hardware... No matter what, it always ends up the same.
>
> The process starts well, connects to the database and fetches all the
> data, but then it seems to get stuck after this output line:
> Info: Connecting to mysql2 database: puppet
>
> Meanwhile, a CPU is working and the amount of memory used by this process
> increases constantly. I have tried on several servers, one with up to 28GB
> of RAM, and it died the same way.
>
> Here are the facts of my last attempt:
>
> The execution output:
>
> [root@ST-L226-ROGTORG ~]# time puppet storeconfigs --verbose --debug export
> Debug: Using settings: adding file resource 'confdir':
> 'File[/etc/puppet]{:path=>"/etc/puppet", :ensure=>:directory,
> :loglevel=>:debug, :links=>:follow, :backup=>false}'
> Debug: Using settings: adding file resource 'vardir':
> 'File[/var/lib/puppet]{:path=>"/var/lib/puppet", :ensure=>:directory,
> :loglevel=>:debug, :links=>:follow, :backup=>false}'
> Debug: Failed to load library 'ldap' for feature 'ldap'
> Debug: Puppet::Type::User::ProviderLdap: feature ldap is missing
> Debug: Puppet::Type::User::ProviderDirectoryservice: file
> /usr/bin/dsimport does not exist
> Debug: Puppet::Type::User::ProviderPw: file pw does not exist
> Debug: Puppet::Type::User::ProviderUser_role_add: file roleadd does not
> exist
> Debug: Failed to load library 'ldap' for feature 'ldap'
> Debug: Puppet::Type::Group::ProviderLdap: feature ldap is missing
> Debug: Puppet::Type::Group::ProviderDirectoryservice: file /usr/bin/dscl
> does not exist
> Debug: Puppet::Type::Group::ProviderPw: file pw does not exist
> Debug: Using settings: adding file resource 'logdir':
> 'File[/var/log/puppet]{:path=>"/var/log/puppet", :mode=>"750",
> :owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug,
> :links=>:follow, :backup=>false}'
> Debug: Using settings: adding file resource 'statedir':
> 'File[/var/lib/puppet/state]{:path=>"/var/lib/puppet/state", :mode=>"1755",
> :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
> Debug: Using settings: adding file resource 'rundir':
> 'File[/var/run/puppet]{:path=>"/var/run/puppet", :mode=>"755",
> :owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug,
> :links=>:follow, :backup=>false}'
> Debug: Using settings: adding file resource 'libdir':
> 'File[/var/lib/puppet/lib]{:path=>"/var/lib/puppet/lib",
> :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
> Debug: Using settings: adding file resource 'certdir':
> 'File[/var/lib/puppet/ssl/certs]{:path=>"/var/lib/puppet/ssl/certs",
> :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow,
> :backup=>false}'
> Debug: Using settings: adding file resource 'ssldir':
> 'File[/var/lib/puppet/ssl]{:path=>"/var/lib/puppet/ssl", :mode=>"771",
> :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow,
> :backup=>false}'
> Debug: Using settings: adding file resource 'publickeydir':
> 'File[/var/lib/puppet/ssl/public_keys]{:path=>"/var/lib/puppet/ssl/public_keys",
>
> :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow,
> :backup=>false}'
> Debug: Using settings: adding file resource 'requestdir':
> 'File[/var/lib/puppet/ssl/certificate_requests]{:path=>"/var/lib/puppet/ssl/certificate_requests",
>
> :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow,
> :backup=>false}'
> Debug: Using settings: adding file resource 'privatekeydir':
> 'File[/var/lib/puppet/ssl/private_keys]{:path=>"/var/lib/puppet/ssl/private_keys",
>
> :mode=>"750", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug,
> :links=>:follow, :backup=>false}'
> Debug: Using settings: adding file resource 'privatedir':
> 'File[/var/lib/puppet/ssl/private]{:path=>"/var/lib/puppet/ssl/private",
> :mode=>"750", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug,
> :links=>:follow, :backup=>false}'
> Debug: Using settings: adding file resource 'manifestdir':
> 'File[/etc/puppet/manifests]{:path=>"/etc/puppet/manifests",
> :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
> Debug: Using settings: adding file resource 'masterhttplog':
> 'File[/var/log/puppet/masterhttp.log]{:path=>"/var/log/puppet/masterhttp.log",
>
> :mode=>"660", :owner=>"puppet", :group=>"puppet", :ensure=>:file,
> :loglevel=>:debug, :links=>:follow, :backup=>false}'
> Debug: Using settings: adding file resource 'bucketdir':
> 'File[/var/lib/puppet/bucket]{:path=>"/var/lib/puppet/bucket",
> :mode=>"750", :owner=>"puppet", :group=>"puppet", :ensure=>:directory,
> :loglevel=>:debug, :links=>:follow, :backup=>false}'
> Debug: Using settings: adding file resource 'rest_authconfig':
> 'File[/etc/puppet/auth.conf]{:path=>"/etc/puppet/auth.conf",
> :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'
> Debug: Using settings: adding file resource 'yamldir':
> 'File[/var/lib/puppet/yaml]{:path=>"/var/lib/puppet/yaml", :mode=>"750",
> :owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug,
> :links=>:follow, :backup=>false}'
> Debug: Using settings: adding file resource 'server_datadir':
> 'File[/var/lib/puppet/server_data]{:path=>"/var/lib/puppet/server_data",
> :mode=>"750", :owner=>"puppet", :group=>"puppet", :ensure=>:directory,
> :loglevel=>:debug, :links=>:follow, :backup=>false}'
> Debug: Using settings: adding file resource 'reportdir':
> 'File[/var/lib/puppet/reports]{:path=>"/var/lib/puppet/reports",
> :mode=>"750", :owner=>"puppet", :group=>"puppet", :ensure=>:directory,
> :loglevel=>:debug, :links=>:follow, :backup=>false}'
> Debug: Using settings: adding file resource 'railslog':
> 'File[/var/log/puppet/rails.log]{:path=>"/var/log/puppet/rails.log",
> :mode=>"600", :owner=>"puppet", :group=>"puppet", :ensure=>:file,
> :loglevel=>:debug, :links=>:follow, :backup=>false}'
> Debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet]
> Debug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet]
> Debug: /File[/var/lib/puppet/ssl/certs]: Autorequiring
> File[/var/lib/puppet/ssl]
> Debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet]
> Debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring
> File[/var/lib/puppet/ssl]
> Debug: /File[/var/lib/puppet/ssl/certificate_requests]: Autorequiring
> File[/var/lib/puppet/ssl]
> Debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring
> File[/var/lib/puppet/ssl]
> Debug: /File[/var/lib/puppet/ssl/private]: Autorequiring
> File[/var/lib/puppet/ssl]
> Debug: /File[/etc/puppet/manifests]: Autorequiring File[/etc/puppet]
> Debug: /File[/var/log/puppet/masterhttp.log]: Autorequiring
> File[/var/log/puppet]
> Debug: /File[/var/lib/puppet/bucket]: Autorequiring File[/var/lib/puppet]
> Debug: /File[/etc/puppet/auth.conf]: Autorequiring File[/etc/puppet]
> Debug: /File[/var/lib/puppet/yaml]: Autorequiring File[/var/lib/puppet]
> Debug: /File[/var/lib/puppet/server_data]: Autorequiring
> File[/var/lib/puppet]
> Debug: /File[/var/lib/puppet/reports]: Autorequiring File[/var/lib/puppet]
> Debug: /File[/var/log/puppet/rails.log]: Autorequiring
> File[/var/log/puppet]
> Debug: Finishing transaction 34502460
> Info: Connecting to mysql2 database: puppet
> Killed
>
> real 66m35.810s
> user 9m42.143s
> sys 6m31.294s
>
>
> My /etc/puppet/puppet.conf file:
>
> [main]
> # The Puppet log directory.
> # The default value is '$vardir/log'.
> logdir = /var/log/puppet
>
> # Where Puppet PID files are kept.
> # The default value is '$vardir/run'.
> rundir = /var/run/puppet
>
> # Where SSL certificates are kept.
> # The default value is '$confdir/ssl'.
> ssldir = $vardir/ssl
>
> facts_terminus = inventory_active_record
>
> storeconfigs = true
> dbadapter = mysql2
> dbserver = roger-test.ofi.softonic.lan
> dbname = puppet
> dbuser = puppet
> dbpassword =
> dbconnections = 20
>
> [agent]
> # The file in which puppetd stores a list of the classes
> # associated with the retrieved configuratiion. Can be loaded in
> # the separate ``puppet`` executable using the ``--loadclasses``
> # option.
> # The default value is '$confdir/classes.txt'.
> classfile = $vardir/classes.txt
>
> # Where puppetd caches the local configuration. An
> # extension indicating the cache format is added automatically.
> # The default value is '$confdir/localconfig'.
> localconfig = $vardir/localconfig
>
> Packages:
>
> [root@ST-L226-ROGTORG ~]# rpm -qa | egrep -i "(puppet|ruby)" | sort
> libselinux-ruby-2.1.13-15.fc19.x86_64
> puppet-3.3.1-1.fc19.noarch
> puppetdb-terminus-1.5.0-1.fc19.noarch
> puppetlabs-release-19-2.noarch
> ruby-2.0.0.247-15.fc19.x86_64
> ruby-augeas-0.5.0-1.fc19.x86_64
> ruby-devel-2.0.0.247-15.fc19.x86_64
> ruby-irb-2.0.0.247-15.fc19.noarch
> ruby-libs-2.0.0.247-15.fc19.x86_64
> ruby-rgen-0.6.5-1.fc19.noarch
> ruby-shadow-1.4.1-19.fc19.x86_64
> rubygem-activemodel-3.2.13-1.fc19.noarch
> rubygem-activerecord-3.2.13-1.fc19.noarch
> rubygem-activesupport-3.2.13-1.fc19.noarch
> rubygem-arel-3.0.2-3.fc19.noarch
> rubygem-bigdecimal-1.2.0-15.fc19.x86_64
> rubygem-builder-3.1.4-3.fc19.noarch
> rubygem-i18n-0.6.1-3.fc19.noarch
> rubygem-io-console-0.4.2-15.fc19.x86_64
> rubygem-json-1.7.7-100.fc19.x86_64
> rubygem-multi_json-1.7.1-1.fc19.noarch
> rubygem-mysql2-0.3.13-1.fc19.x86_64
> rubygem-psych-2.0.0-15.fc19.x86_64
> rubygem-rdoc-4.0.1-1.fc19.noarch
> rubygems-2.0.10-106.fc19.noarch
> rubygem-tzinfo-0.3.35-1.fc19.noarch
> rubypick-1.1.0-1.fc19.noarch
>
>
> I'm positive the MySQL server receives the query and returns the data,
> according to MySQL's slow-queries.log:
>
> # Time: 131008 17:06:14
> # User@Host: puppet[puppet] @ [172.20.59.127]
> # Thread_id: 169 Schema: puppet Last_errno: 0 Killed: 0
> # Query_time: 189.509563 Lock_time: 0.000136 Rows_sent: 6168111
> Rows_examined: 12712154 Rows_affected: 0 Rows_read: 18446744073709551613
> # Bytes_sent: 2426331478 Tmp_tables: 0 Tmp_disk_tables: 0
> Tmp_table_sizes: 0
> # InnoDB_trx_id: A93
> SET timestamp=1381244774;
> SELECT `hosts`.`id` AS t0_r0, `hosts`.`name` AS t0_r1, `hosts`.`ip` AS
> t0_r2, `hosts`.`environment` AS t0_r3, `hosts`.`last_compile` AS t0_r4,
> `hosts`.`last_freshcheck` AS t0_r5, `
> hosts`.`last_report` AS t0_r6, `hosts`.`updated_at` AS t0_r7,
> `hosts`.`source_file_id` AS t0_r8, `hosts`.`created_at` AS t0_r9,
> `resources`.`id` AS t1_r0, `resources`.`title` AS t1
> _r1, `resources`.`restype` AS t1_r2, `resources`.`host_id` AS t1_r3,
> `resources`.`source_file_id` AS t1_r4, `resources`.`exported` AS t1_r5,
> `resources`.`line` AS t1_r6, `resources
> `.`updated_at` AS t1_r7, `resources`.`created_at` AS t1_r8,
> `param_values`.`id` AS t2_r0, `param_values`.`value` AS t2_r1,
> `param_values`.`param_name_id` AS t2_r2, `param_values`.`
> line` AS t2_r3, `param_values`.`resource_id` AS t2_r4,
> `param_values`.`updated_at` AS t2_r5, `param_values`.`created_at` AS t2_r6,
> `puppet_tags`.`id` AS t3_r0, `puppet_tags`.`name`
> AS t3_r1, `puppet_tags`.`updated_at` AS t3_r2, `puppet_tags`.`created_at`
> AS t3_r3 FROM `hosts` LEFT OUTER JOIN `resources` ON `resources`.`host_id`
> = `hosts`.`id` LEFT OUTER JOIN
> `param_values` ON `param_values`.`resource_id` = `resources`.`id` LEFT
> OUTER JOIN `resource_tags` ON `resource_tags`.`resource_id` =
> `resources`.`id` LEFT OUTER JOIN `puppet_tags`
> ON `puppet_tags`.`id` = `resource_tags`.`puppet_tag_id` WHERE
> `resources`.`exported` = 1;
>
>
> When the process gets stuck, an strace shows a constant and very quick
> flow of:
>
> 30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
>
> The final moments of the strace showed:
>
> 30667 restart_syscall(<... resuming interrupted call ...> <unfinished ...>
> 30666 mremap(0x7f7aa4d0a000, 2732032, 2752512, MREMAP_MAYMOVE) =
> 0x7f7aa4d0a000
> 30666 brk(0) = 0x23aaea000
> 30666 brk(0x23ab0b000) = 0x23ab0b000
> 30666 brk(0) = 0x23ab0b000
> 30666 brk(0x23ab2c000) = 0x23ab2c000
> 30666 brk(0) = 0x23ab2c000
> 30666 brk(0x23ab4d000) = 0x23ab4d000
> ... then lots of...
> 30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
> 30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
> 30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
> ... and finally ...
> 30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
> 30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
> 30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
> 30667 +++ killed by SIGKILL +++
> 30666 +++ killed by SIGKILL +++
>
> Any idea of what is happening? Am I doing something wrong?
>
> Thanks!
>
> Roger Torrentsgenerós
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.
