Issue #13038 has been updated by Erik Dalén.
More easily read output:
strace:
strace -p 15272
Process 15272 attached - interrupt to quit
select(0, NULL, NULL, NULL, {0, 386506}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}^C <unfinished ...>
Process 15272 detached
lsof:
# lsof -p 15272
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
puppet 15272 root cwd DIR 254,0 4096 917506 /root
puppet 15272 root rtd DIR 254,0 4096 2 /
puppet 15272 root txt REG 254,0 5424 794665
/usr/bin/ruby1.8
puppet 15272 root mem REG 254,0 80712 1180151
/lib/libresolv-2.11.2.so
puppet 15272 root mem REG 254,0 22928 1180172
/lib/libnss_dns-2.11.2.so
puppet 15272 root mem REG 254,0 37984 922400
/usr/lib/ruby/1.8/x86_64-linux/zlib.so
puppet 15272 root mem REG 254,0 10784 925695
/usr/lib/ruby/1.8/x86_64-linux/shadow.so
puppet 15272 root mem REG 254,0 8648 922394
/usr/lib/ruby/1.8/x86_64-linux/digest/md5.so
puppet 15272 root mem REG 254,0 45824 922412
/usr/lib/ruby/1.8/x86_64-linux/socket.so
puppet 15272 root mem REG 254,0 93936 793016
/usr/lib/libz.so.1.2.3.4
puppet 15272 root mem REG 254,0 1693344 797851
/usr/lib/libcrypto.so.0.9.8
puppet 15272 root mem REG 254,0 349248 797852
/usr/lib/libssl.so.0.9.8
puppet 15272 root mem REG 254,0 298664 922401
/usr/lib/ruby/1.8/x86_64-linux/openssl.so
puppet 15272 root mem REG 254,0 15304 922415
/usr/lib/ruby/1.8/x86_64-linux/racc/cparse.so
puppet 15272 root mem REG 254,0 12712 922397
/usr/lib/ruby/1.8/x86_64-linux/digest.so
puppet 15272 root mem REG 254,0 12808 922392
/usr/lib/ruby/1.8/x86_64-linux/digest/sha1.so
puppet 15272 root mem REG 254,0 14392 791358
/usr/lib/gconv/UTF-16.so
puppet 15272 root mem REG 254,0 17064 922409
/usr/lib/ruby/1.8/x86_64-linux/iconv.so
puppet 15272 root mem REG 254,0 19352 922402
/usr/lib/ruby/1.8/x86_64-linux/strscan.so
puppet 15272 root mem REG 254,0 271400 922421
/usr/lib/ruby/1.8/x86_64-linux/nkf.so
puppet 15272 root mem REG 254,0 12680 922406
/usr/lib/ruby/1.8/x86_64-linux/syslog.so
puppet 15272 root mem REG 254,0 47616 1180155
/lib/libnss_files-2.11.2.so
puppet 15272 root mem REG 254,0 43552 1180152
/lib/libnss_nis-2.11.2.so
puppet 15272 root mem REG 254,0 89064 1180168
/lib/libnsl-2.11.2.so
puppet 15272 root mem REG 254,0 31616 1180165
/lib/libnss_compat-2.11.2.so
puppet 15272 root mem REG 254,0 5344 922420
/usr/lib/ruby/1.8/x86_64-linux/fcntl.so
puppet 15272 root mem REG 254,0 11032 922399
/usr/lib/ruby/1.8/x86_64-linux/etc.so
puppet 15272 root mem REG 254,0 16560 922398
/usr/lib/ruby/1.8/x86_64-linux/thread.so
puppet 15272 root mem REG 254,0 119984 922403
/usr/lib/ruby/1.8/x86_64-linux/syck.so
puppet 15272 root mem REG 254,0 21120 922413
/usr/lib/ruby/1.8/x86_64-linux/stringio.so
puppet 15272 root mem REG 254,0 1432968 1180158
/lib/libc-2.11.2.so
puppet 15272 root mem REG 254,0 530736 1180167
/lib/libm-2.11.2.so
puppet 15272 root mem REG 254,0 35104 1180169
/lib/libcrypt-2.11.2.so
puppet 15272 root mem REG 254,0 14696 1180157
/lib/libdl-2.11.2.so
puppet 15272 root mem REG 254,0 31744 1180160
/lib/librt-2.11.2.so
puppet 15272 root mem REG 254,0 131260 1180159
/lib/libpthread-2.11.2.so
puppet 15272 root mem REG 254,0 927840 794548
/usr/lib/libruby1.8.so.1.8.7
puppet 15272 root mem REG 254,0 128744 1180170
/lib/ld-2.11.2.so
puppet 15272 root mem REG 254,0 26048 793220
/usr/lib/gconv/gconv-modules.cache
puppet 15272 root 0r FIFO 0,8 0t0 765633332 pipe
puppet 15272 root 1u REG 254,0 0 393323
/tmp/tmpfs2dR5B (deleted)
puppet 15272 root 2u REG 254,0 0 393323
/tmp/tmpfs2dR5B (deleted)
puppet 15272 root 3r FIFO 0,8 0t0 765635635 pipe
puppet 15272 root 4w FIFO 0,8 0t0 765635635 pipe
puppet 15272 root 5u unix 0xffff8801b88f0f00 0t0 765635637 socket
----------------------------------------
Bug #13038: Hung agent processes
https://projects.puppetlabs.com/issues/13038#change-56427
Author: Erik Dalén
Status: Unreviewed
Priority: Normal
Assignee:
Category:
Target version:
Affected Puppet version: 2.7.11
Keywords: select hung process stuck
Branch:
Since upgrading to 2.7.11 (from 2.6.2) we have experienced a number or hosts
where the puppet process has gotten stuck. Doing a strace shows the following:
Process 15272 attached - interrupt to quit
select(0, NULL, NULL, NULL, {0, 386506}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}^C <unfinished ...>
Process 15272 detached
lsof output:
# lsof -p 15272
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
puppet 15272 root cwd DIR 254,0 4096 917506 /root
puppet 15272 root rtd DIR 254,0 4096 2 /
puppet 15272 root txt REG 254,0 5424 794665
/usr/bin/ruby1.8
puppet 15272 root mem REG 254,0 80712 1180151
/lib/libresolv-2.11.2.so
puppet 15272 root mem REG 254,0 22928 1180172
/lib/libnss_dns-2.11.2.so
puppet 15272 root mem REG 254,0 37984 922400
/usr/lib/ruby/1.8/x86_64-linux/zlib.so
puppet 15272 root mem REG 254,0 10784 925695
/usr/lib/ruby/1.8/x86_64-linux/shadow.so
puppet 15272 root mem REG 254,0 8648 922394
/usr/lib/ruby/1.8/x86_64-linux/digest/md5.so
puppet 15272 root mem REG 254,0 45824 922412
/usr/lib/ruby/1.8/x86_64-linux/socket.so
puppet 15272 root mem REG 254,0 93936 793016
/usr/lib/libz.so.1.2.3.4
puppet 15272 root mem REG 254,0 1693344 797851
/usr/lib/libcrypto.so.0.9.8
puppet 15272 root mem REG 254,0 349248 797852
/usr/lib/libssl.so.0.9.8
puppet 15272 root mem REG 254,0 298664 922401
/usr/lib/ruby/1.8/x86_64-linux/openssl.so
puppet 15272 root mem REG 254,0 15304 922415
/usr/lib/ruby/1.8/x86_64-linux/racc/cparse.so
puppet 15272 root mem REG 254,0 12712 922397
/usr/lib/ruby/1.8/x86_64-linux/digest.so
puppet 15272 root mem REG 254,0 12808 922392
/usr/lib/ruby/1.8/x86_64-linux/digest/sha1.so
puppet 15272 root mem REG 254,0 14392 791358
/usr/lib/gconv/UTF-16.so
puppet 15272 root mem REG 254,0 17064 922409
/usr/lib/ruby/1.8/x86_64-linux/iconv.so
puppet 15272 root mem REG 254,0 19352 922402
/usr/lib/ruby/1.8/x86_64-linux/strscan.so
puppet 15272 root mem REG 254,0 271400 922421
/usr/lib/ruby/1.8/x86_64-linux/nkf.so
puppet 15272 root mem REG 254,0 12680 922406
/usr/lib/ruby/1.8/x86_64-linux/syslog.so
puppet 15272 root mem REG 254,0 47616 1180155
/lib/libnss_files-2.11.2.so
puppet 15272 root mem REG 254,0 43552 1180152
/lib/libnss_nis-2.11.2.so
puppet 15272 root mem REG 254,0 89064 1180168
/lib/libnsl-2.11.2.so
puppet 15272 root mem REG 254,0 31616 1180165
/lib/libnss_compat-2.11.2.so
puppet 15272 root mem REG 254,0 5344 922420
/usr/lib/ruby/1.8/x86_64-linux/fcntl.so
puppet 15272 root mem REG 254,0 11032 922399
/usr/lib/ruby/1.8/x86_64-linux/etc.so
puppet 15272 root mem REG 254,0 16560 922398
/usr/lib/ruby/1.8/x86_64-linux/thread.so
puppet 15272 root mem REG 254,0 119984 922403
/usr/lib/ruby/1.8/x86_64-linux/syck.so
puppet 15272 root mem REG 254,0 21120 922413
/usr/lib/ruby/1.8/x86_64-linux/stringio.so
puppet 15272 root mem REG 254,0 1432968 1180158
/lib/libc-2.11.2.so
puppet 15272 root mem REG 254,0 530736 1180167
/lib/libm-2.11.2.so
puppet 15272 root mem REG 254,0 35104 1180169
/lib/libcrypt-2.11.2.so
puppet 15272 root mem REG 254,0 14696 1180157
/lib/libdl-2.11.2.so
puppet 15272 root mem REG 254,0 31744 1180160
/lib/librt-2.11.2.so
puppet 15272 root mem REG 254,0 131260 1180159
/lib/libpthread-2.11.2.so
puppet 15272 root mem REG 254,0 927840 794548
/usr/lib/libruby1.8.so.1.8.7
puppet 15272 root mem REG 254,0 128744 1180170
/lib/ld-2.11.2.so
puppet 15272 root mem REG 254,0 26048 793220
/usr/lib/gconv/gconv-modules.cache
puppet 15272 root 0r FIFO 0,8 0t0 765633332 pipe
puppet 15272 root 1u REG 254,0 0 393323
/tmp/tmpfs2dR5B (deleted)
puppet 15272 root 2u REG 254,0 0 393323
/tmp/tmpfs2dR5B (deleted)
puppet 15272 root 3r FIFO 0,8 0t0 765635635 pipe
puppet 15272 root 4w FIFO 0,8 0t0 765635635 pipe
puppet 15272 root 5u unix 0xffff8801b88f0f00 0t0 765635637 socket
--
You have received this notification because you have either subscribed to it,
or are involved in it.
To change your notification preferences, please click here:
http://projects.puppetlabs.com/my/account
--
You received this message because you are subscribed to the Google Groups
"Puppet Bugs" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/puppet-bugs?hl=en.
