Jira (PUP-8663) http_keepalive_timeout setting doesn't work correctly under Ruby 2.x
Title: Message Title Josh Cooper updated an issue Puppet / PUP-8663 http_keepalive_timeout setting doesn't work correctly under Ruby 2.x Change By: Josh Cooper Release Notes Summary: Puppet on ruby 2.0 or greater would close and reopen HTTP connection that were idle for more than 2 seconds, causing increased load on puppetservers. This change ensures the agent always uses our `http_keepalive_timeout` setting when determining when to close idle connections. Release Notes: Bug Fix Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8663) http_keepalive_timeout setting doesn't work correctly under Ruby 2.x
Title: Message Title Kenn Hussey commented on PUP-8663 Re: http_keepalive_timeout setting doesn't work correctly under Ruby 2.x Josh Cooper please add release notes for this issue if needed, thanks! Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8663) http_keepalive_timeout setting doesn't work correctly under Ruby 2.x
Title: Message Title Josh Cooper updated an issue Puppet / PUP-8663 http_keepalive_timeout setting doesn't work correctly under Ruby 2.x Change By: Josh Cooper Fix Version/s: PUP 5.3.z Fix Version/s: PUP 4.10.z Fix Version/s: PUP 5.3.7 Fix Version/s: PUP 4.10.12 Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com
Jira (PUP-8663) http_keepalive_timeout setting doesn't work correctly under Ruby 2.x
Title: Message Title Josh Cooper commented on PUP-8663 Re: http_keepalive_timeout setting doesn't work correctly under Ruby 2.x Thanks, I submitted PRs to merge up to 5.3.x and 5.5.x, and then it'll get automerged to master. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8663) http_keepalive_timeout setting doesn't work correctly under Ruby 2.x
Title: Message Title Jayant Sane commented on PUP-8663 Re: http_keepalive_timeout setting doesn't work correctly under Ruby 2.x Disregard if you were already planning to do the needful ... but per Sean's email we might see a CI on 1.10.x PA and neither automatic merge-ups. You may want to merge this to 5.5.x assuming it is meeting the criteria to go to 1.10.x or 5.3.x. thx Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8663) http_keepalive_timeout setting doesn't work correctly under Ruby 2.x
Title: Message Title Jacob Helwig commented on PUP-8663 Re: http_keepalive_timeout setting doesn't work correctly under Ruby 2.x Merged to 4.10.x in af6f87ae55. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8663) http_keepalive_timeout setting doesn't work correctly under Ruby 2.x
Title: Message Title Craig Gomes assigned an issue to Jacob Helwig Puppet / PUP-8663 http_keepalive_timeout setting doesn't work correctly under Ruby 2.x Change By: Craig Gomes Assignee: Josh Cooper Jacob Helwig Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8663) http_keepalive_timeout setting doesn't work correctly under Ruby 2.x
Title: Message Title Josh Cooper assigned an issue to Josh Cooper Puppet / PUP-8663 http_keepalive_timeout setting doesn't work correctly under Ruby 2.x Change By: Josh Cooper Assignee: Josh Cooper Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8663) http_keepalive_timeout setting doesn't work correctly under Ruby 2.x
Title: Message Title Josh Cooper updated an issue Puppet / PUP-8663 http_keepalive_timeout setting doesn't work correctly under Ruby 2.x Change By: Josh Cooper Sprint: Platform Core KANBAN Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8663) http_keepalive_timeout setting doesn't work correctly under Ruby 2.x
Title: Message Title Josh Cooper updated an issue Puppet / PUP-8663 http_keepalive_timeout setting doesn't work correctly under Ruby 2.x Change By: Josh Cooper Fix Version/s: PUP 5.5.2 Fix Version/s: PUP 5.3.z Fix Version/s: PUP 4.10.z Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8663) http_keepalive_timeout setting doesn't work correctly under Ruby 2.x
Title: Message Title Owen Rodabaugh updated an issue Puppet / PUP-8663 http_keepalive_timeout setting doesn't work correctly under Ruby 2.x Change By: Owen Rodabaugh CS Priority: Needs Priority Major CS Impact: When we introduced puppetserver one of the gains was from connection reuse, this bug would seem to indicate that in some situations connections may not be getting reused because the ruby agent times out too soon.Support has been seeing mysteriously high numbers of connections which are believe to be related this issue. We're unsure if this may also be impacting intermediary network devices in situations with a large number of agents that are no longer reusing connections. CS Severity: 2 - Annoyance CS Business Value: 4 - $ CS Frequency: 5 - >90% of Customers Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Jira (PUP-8663) http_keepalive_timeout setting doesn't work correctly under Ruby 2.x
Title: Message Title Nick Walker updated an issue Puppet / PUP-8663 http_keepalive_timeout setting doesn't work correctly under Ruby 2.x Change By: Nick Walker CS Priority: Needs Priority Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8663) http_keepalive_timeout setting doesn't work correctly under Ruby 2.x
Title: Message Title Josh Cooper updated an issue Puppet / PUP-8663 http_keepalive_timeout setting doesn't work correctly under Ruby 2.x Change By: Josh Cooper Team: Platform Core Coremunity Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8663) http_keepalive_timeout setting doesn't work correctly under Ruby 2.x
Title: Message Title
Charlie Sharpsteen updated an issue
Puppet / PUP-8663
http_keepalive_timeout setting doesn't work correctly under Ruby 2.x
Change By:
Charlie Sharpsteen
Back when the {{http_keepalive_timeout}} setting was introduced, Puppet ran under Ruby 1.9.3 and life was good:https://github.com/puppetlabs/puppet/blob/5.5.0/lib/puppet/network/http/pool.rbThen Ruby 2.x descended from the mountains and brought with it a {{Net::HTTP#keep_alive_timeout}}:https://github.com/ruby/ruby/blob/v2_4_3/lib/net/http.rb#L664Ruby's timeout defaults to 2 seconds and always overrides the timeout used by the Puppet agent. This causes HTTP connections to be closed early instead of being re-used which increases the TLS handshake load on Puppet Server along with the amount of connection state that network devices between the agent and server have to contend with.h2. Reproduction Case - Install {{puppet-agent}} 5.5.0 and {{puppetserver}} 5.3.0 on a CentOS 7node:{code:bash}rpm -Uvh http://yum.puppetlabs.com/puppet5/puppet5-release-el-7.noarch.rpmyum install -y puppet-agent-5.5.0 puppetserver-5.3.0{code} - Configure the puppet agent to connect to the local host and startpuppetserver:{code:bash}/opt/puppetlabs/bin/puppet config set --section main server $(hostname -f)systemctl start puppetserver{code} - Run {{puppet agent -t}} to verify things work. - Add a custom fact that will sleep for 4 seconds, which delays theHTTP POST request for a catalog:{code:bash}cat <<-'EOF' > /etc/puppetlabs/code/modules/testmod/lib/facter/sleep_fact.rbFacter.add(:sleep_fact) do setcode do$stdout.puts("Sleeping for 4 seconds...")Kernel.sleep(4)nil endendEOF{code} - Set Puppet's {{http_keepalive_timeout}} to 15 seconds:{code:bash}/opt/puppetlabs/bin/puppet config set --section main http_keepalive_timeout 15s{code} - Run {{puppet agent -t}} and track HTTP connections:{code:bash}/opt/puppetlabs/bin/puppet agent -t --http_debug 2>&1|grep -v '^->\|<-'{code}h3. OutcomeThe Puppet Agent opens a new TCP connection after loading facts despite{{http_keepalive_timeout}} being set to 15 seconds:{noformat}# /opt/puppetlabs/bin/puppet agent -t --http_debug 2>&1|grep -v '^->\|<-'opening connection to m4t83ynafhampi9.delivery.puppetlabs.net:8140...openedstarting SSL for m4t83ynafhampi9.delivery.puppetlabs.net:8140...SSL establishedreading 3339 bytes...read 3339 bytesConn keep-aliveInfo: Using configured environment 'production'Info: Retrieving pluginfactsreading 204 bytes...read 204 bytesConn keep-aliveInfo: Retrieving pluginreading 259 bytes...read 259 bytesConn keep-aliveInfo: Retrieving localesreading 204 bytes...read 204 bytesConn keep-alive Info: Loading factsSleeping for 4 seconds...Conn close because of keep_alive_timeoutopening connection to m4t83ynafhampi9.delivery.puppetlabs.net:8140...openedstarting SSL for m4t83ynafhampi9.delivery.puppetlabs.net:8140...SSL established reading 310 bytes...read 310 bytesConn keep-aliveInfo: Caching catalog for m4t83ynafhampi9.delivery.puppetlabs.netInfo: Applying configu
Jira (PUP-8663) http_keepalive_timeout setting doesn't work correctly under Ruby 2.x
Title: Message Title
Charlie Sharpsteen updated an issue
Puppet / PUP-8663
http_keepalive_timeout setting doesn't work correctly under Ruby 2.x
Change By:
Charlie Sharpsteen
Back when the {{http_keepalive_timeout}} setting was introduced, Puppet ran under Ruby 1.9.3 and life was good :https://github . com/puppetlabs/puppet/blob/5.5.0/lib/puppet/network/http/pool.rb Then Ruby 2.x descended from the mountains and brought with it a {{Net::HTTP#keep_alive_timeout}} :https://github . com/ruby/ruby/blob/v2_4_3/lib/net/http.rb#L664 Ruby's timeout defaults to 2 seconds and always overrides the timeout used by the Puppet agent. This causes HTTP connections to be closed early instead of being re-used which increases the TLS handshake load on Puppet Server along with the amount of connection state that network devices between the agent and server have to contend with. h2. Reproduction Case - Install {{puppet-agent}} 5.5.0 and {{puppetserver}} 5.3.0 on a CentOS 7node:{code:bash}rpm -Uvh http://yum.puppetlabs.com/puppet5/puppet5-release-el-7.noarch.rpmyum install -y puppet-agent-5.5.0 puppetserver-5.3.0{code} - Configure the puppet agent to connect to the local host and startpuppetserver:{code:bash}/opt/puppetlabs/bin/puppet config set --section main server $(hostname -f)systemctl start puppetserver{code} - Run {{puppet agent -t}} to verify things work. - Add a custom fact that will sleep for 4 seconds, which delays theHTTP POST request for a catalog:{code:bash}cat <<-'EOF' > /etc/puppetlabs/code/modules/testmod/lib/facter/sleep_fact.rbFacter.add(:sleep_fact) do setcode do$stdout.puts("Sleeping for 4 seconds...")Kernel.sleep(4)nil endendEOF{code} - Set Puppet's {{http_keepalive_timeout}} to 15 seconds:{code:bash}/opt/puppetlabs/bin/puppet config set --section main http_keepalive_timeout 15s{code} - Run {{puppet agent -t}} and track HTTP connections:{code:bash}/opt/puppetlabs/bin/puppet agent -t --http_debug 2>&1|grep -v '^->\|<-'{code}h3. OutcomeThe Puppet Agent opens a new TCP connection after loading facts despite{{http_keepalive_timeout}} being set to 15 seconds:{noformat}# /opt/puppetlabs/bin/puppet agent -t --http_debug 2>&1|grep -v '^->\|<-'opening connection to m4t83ynafhampi9.delivery.puppetlabs.net:8140...openedstarting SSL for m4t83ynafhampi9.delivery.puppetlabs.net:8140...SSL establishedreading 3339 bytes...read 3339 bytesConn keep-aliveInfo: Using configured environment 'production'Info: Retrieving pluginfactsreading 204 bytes...read 204 bytesConn keep-aliveInfo: Retrieving pluginreading 259 bytes...read 259 bytesConn keep-aliveInfo: Retrieving localesreading 204 bytes...read 204 bytesConn keep-aliveInfo: Loading factsSleeping for 4 seconds...Conn close because of keep_alive_timeoutopening connection to m4t83ynafhampi9.delivery.puppetlabs.net:8140...openedstarting SSL for m4t83ynafhampi9.delivery.puppetlabs.net:8140...SSL establishedreading 310 bytes...read 310 bytesConn keep-aliveInfo: Caching catalog for m4t83ynafhampi9.delivery.puppetlabs.netInfo: Apply
Jira (PUP-8663) http_keepalive_timeout setting doesn't work correctly under Ruby 2.x
Title: Message Title Charlie Sharpsteen created an issue Puppet / PUP-8663 http_keepalive_timeout setting doesn't work correctly under Ruby 2.x Issue Type: Bug Affects Versions: PUP 5.5.0, PUP 4.10.10 Assignee: Unassigned Created: 2018/04/13 6:14 PM Priority: Normal Reporter: Charlie Sharpsteen Back when the http_keepalive_timeout setting was introduced, Puppet ran under Ruby 1.9.3 and life was good. Then Ruby 2.x descended from the mountains and brought with it a Net::HTTP#keep_alive_timeout. Ruby's timeout defaults to 2 seconds and always overrides the timeout used by the Puppet agent. Reproduction Case Install puppet-agent 5.5.0 and puppetserver 5.3.0 on a CentOS 7 node: rpm -Uvh http://yum.puppetlabs.com/puppet5/puppet5-release-el-7.noarch.rpm yum install -y puppet-agent-5.5.0 puppetserver-5.3.0 Configure the puppet agent to connect to the local host and start puppetserver:
