Jira (PUP-1680) incorrect header check using Ruby 2.1.0
Title: Message Title
Kyle Smith commented on an issue
Re: incorrect header check using Ruby 2.1.0
I've found the root cause of this issue, but I'm not sure what the preferred way to tackle it is.
It's a bit of a rat's nest, but here's the idea:
In content.rb:
def chunk_file_from_source(source_or_content)
get_from_source(source_or_content) do |response|
case response.code
when /^2/; uncompress(response) { |uncompressor| response.read_body { |chunk| yield uncompressor.uncompress(chunk) } }
else
# Raise the http error if we didn't get a 'success' of some kind.
message = Error #{response.code} on SERVER: #{(response.body||'').empty? ? response.message : uncompress_body(response)}
raise Net::HTTPError.new(message, response)
end
end
end
chunk_file_from_source calls get_from_source with a block, which calls Puppet::Network::HTTP::Connection#request_get, which is a direct wrapper around Net::HTTP#request_get. Ruby 2.1.1 includes all the automatic Accept-Encoding / decompression magic. When you have a response object (yielded in #request_get), it is not yet decompressed. If you check the headers, you will see Content-Encoding: gzip. Once you call #read_body or #read (as is done above in chunk_file_from_source, the decompression and modification of headers is done. Unfortunately the uncompress method of Connection has already read the headers and assigned a ZlibAdapter to decompress the results.
Here's a demo program that can exercise the failure, with comment:
require 'net/http'
require 'openssl'
# Make sure you web server will attempt to compress this file (e.g. mod_deflate)
uri = URI.parse('https://puppet.example.com:8140/devel/file_content/modules/mymodule/myfile.html')
cert = File.read('/var/lib/puppet/ssl/certs/client.example.com.pem')
key = File.read('/var/lib/puppet/ssl/private_keys/client.example.com.pem')
conn = Net::HTTP.new(uri.host, uri.port)
conn.use_ssl = true
conn.cert = OpenSSL::X509::Certificate.new(cert)
conn.key = OpenSSL::PKey::RSA.new(key)
conn.verify_mode = OpenSSL::SSL::VERIFY_NONE
# Add 'Accept-Encoding' = 'None' to the hash to see the inverse behavior.
conn.request_get(uri.path, 'Accept' = 'raw') do |resp|
pp resp.to_hash # Shows the Content-encoding: gzip header.
# Puppet uses this response value to detemine if it needs
# to decompress data.
# Ruby 2.1.1 responds to #decode_content, which does not actually decode any
# content, but instead tells the caller if the content would be decoded when
# a read method is called.
puts resp#decode_content = #{resp.decode_content}
pp resp.body # Triggers Ruby to perform decompression and modify headers.
pp resp.to_hash # Does not show the Content-encoding: gzip header.
end
Jira (PUP-1680) incorrect header check using Ruby 2.1.0
Title: Message Title
Kyle Smith commented on an issue
Re: incorrect header check using Ruby 2.1.0
If anyone else is encountering this issue, I've just confirmed a workaround:
You can set http_compression = true in the [main] section of your puppet.conf file. This tells Puppet to append it's own Accept-Encoding: header to the request, thereby causing Ruby to assume Puppet will handle any necessary decoding (which it will).
The core issue is that in Ruby 2.1.1, Ruby itself tacks an Accept-Encoding: header to every request.
Add Comment
Puppet / PUP-1680
incorrect header check using Ruby 2.1.0
I haven't found anything out there so far, so I thought I'd just document it while I'm at it. We ran into some other problem with the changes in net/http Ruby 2.1.0 introduced around the handling of gzip/deflate encoded content. This MIGHT be related. So far we haven't been able to fix it. {code} Error: incorrect header check Error: /Stage[main]/...
This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede)
--
You received this message because you are subscribed to the Google
Jira (PUP-1680) incorrect header check using Ruby 2.1.0
Title: Message Title Kyle Smith commented on an issue Re: incorrect header check using Ruby 2.1.0 I work with Marc Seeger, and just got this to happen with --trace: Wrapped exception: incorrect header check /usr/local/lib/ruby/gems/2.1.0/gems/puppet-3.5.1/lib/puppet/network/http/compression.rb:69:in `inflate' /usr/local/lib/ruby/gems/2.1.0/gems/puppet-3.5.1/lib/puppet/network/http/compression.rb:69:in `uncompress' /usr/local/lib/ruby/gems/2.1.0/gems/puppet-3.5.1/lib/puppet/type/file/content.rb:221:in `block (3 levels) in chunk_file_from_source' /usr/local/lib/ruby/2.1.0/net/protocol.rb:407:in `call_block' /usr/local/lib/ruby/2.1.0/net/protocol.rb:398:in `' /usr/local/lib/ruby/2.1.0/net/http/response.rb:369:in `block (2 levels) in inflate_adapter' /usr/local/lib/ruby/2.1.0/net/http/response.rb:368:in `inflate' /usr/local/lib/ruby/2.1.0/net/http/response.rb:368:in `block in inflate_adapter' /usr/local/lib/ruby/2.1.0/net/protocol.rb:407:in `call_block' /usr/local/lib/ruby/2.1.0/net/protocol.rb:398:in `' /usr/local/lib/ruby/2.1.0/net/protocol.rb:106:in `read' /usr/local/lib/ruby/2.1.0/net/http/response.rb:390:in `read' /usr/local/lib/ruby/2.1.0/net/http/response.rb:284:in `block in read_body_0' /usr/local/lib/ruby/2.1.0/net/http/response.rb:260:in `inflater' /usr/local/lib/ruby/2.1.0/net/http/response.rb:274:in `read_body_0' /usr/local/lib/ruby/2.1.0/net/http/response.rb:201:in `read_body' /usr/local/lib/ruby/gems/2.1.0/gems/puppet-3.5.1/lib/puppet/type/file/content.rb:221:in `block (2 levels) in chunk_file_from_source' /usr/local/lib/ruby/gems/2.1.0/gems/puppet-3.5.1/lib/puppet/network/http/compression.rb:43:in `uncompress' /usr/local/lib/ruby/gems/2.1.0/gems/puppet-3.5.1/lib/puppet/type/file/content.rb:221:in `block in chunk_file_from_source' /usr/local/lib/ruby/2.1.0/net/http.rb:1415:in `block (2 levels) in transport_request' /usr/local/lib/ruby/2.1.0/net/http/response.rb:162:in `reading_body' /usr/local/lib/ruby/2.1.0/net/http.rb:1414:in `block in transport_request' /usr/local/lib/ruby/2.1.0/net/http.rb:1405:in `catch' /usr/local/lib/ruby/2.1.0/net/http.rb:1405:in `transport_request' /usr/local/lib/ruby/2.1.0/net/http.rb:1378:in `request' /usr/local/lib/ruby/2.1.0/net/http.rb:1371:in `block in request' /usr/local/lib/ruby/2.1.0/net/http.rb:853:in `start' /usr/local/lib/ruby/2.1.0/net/http.rb:1369:in `request' /usr/local/lib/ruby/2.1.0/net/http.rb:1280:in `request_get' /usr/local/lib/ruby/gems/2.1.0/gems/puppet-3.5.1/lib/puppet/network/http/connection.rb:128:in `request_get' /usr/local/lib/ruby/gems/2.1.0/gems/puppet-3.5.1/lib/puppet/type/file/content.rb:213:in `block in get_from_source' /usr/local/lib/ruby/gems/2.1.0/gems/puppet-3.5.1/lib/puppet/indirector/request.rb:258:in `do_request' /usr/local/lib/ruby/gems/2.1.0/gems/puppet-3.5.1/lib/puppet/type/file/content.rb:211:in `get_from_source' /usr/local/lib/ruby/gems/2.1.0/gems/puppet-3.5.1/lib/puppet/type/file/content.rb:219:in `chunk_file_from_source' /usr/local/lib/ruby/gems/2.1.0/gems/puppet-3.5.1/lib/puppet/type/file/content.rb:190:in `each_chunk_from' /usr/local/lib/ruby/gems/2.1.0/gems/puppet-3.5.1/lib/puppet/type/file/content.rb:169:in `block in write' /usr/local/lib/ruby/gems/2.1.0/gems/puppet-3.5.1/lib/puppet/util/checksums.rb:51:in `md5_stream' /usr/local/lib/ruby/gems/2.1.0/gems/puppet-3.5.1/lib/puppet/type/file/checksum.rb:30:in `sum_stream' /usr/local/lib/ruby/gems/2.1.0/gems/puppet-3.5.1/lib/puppet/type/file/content.rb:168:in `write' /usr/local/lib/ruby/gems/2.1.0/gems/puppet-3.5.1/lib/puppet/type/file.rb:854:in `write_content' /usr/local/lib/ruby/gems/2.1.0/gems/puppet-3.5.1/lib/puppet/type/file.rb:758:in `block in write' /usr/local/lib/ruby/gems/2.1.0/gems/puppet-3.5.1/lib/puppet/util.rb:429:in `replace_file'
Jira (PUP-1680) incorrect header check using Ruby 2.1.0
Title: Message Title
Kyle Smith commented on an issue
Re: incorrect header check using Ruby 2.1.0
Found this in compression.rb, not sure what else to look at without actually having the compressed chunk. When Puppet encounters such a failure, it tends to repeat, for what it's worth:
In some local testing an 'incorrect header check' is thrown by Zlib::Inflater#inflate passed invalid data.
def uncompress(chunk)
out = @uncompressor.inflate(chunk)
@first = false
return out
rescue Zlib::DataError
# it can happen that we receive a raw deflate stream
# which might make our inflate throw a data error.
# in this case, we try with a verbatim (no header)
# deflater.
@uncompressor = Zlib::Inflate.new
if @first then
@first = false
retry
end
raise
end
Add Comment
Puppet / PUP-1680
incorrect header check using Ruby 2.1.0
I haven't found anything out there so far, so I thought I'd just document it while I'm at it. We ran into some other problem with the changes in net/http Ruby 2.1.0 introduced around the handling of gzip/deflate encoded content. This MIGHT be related. So far we haven't been able to fix it. {code} Error: incorrect header check Error: /Stage[main]/...
Jira (PUP-1680) incorrect header check using Ruby 2.1.0
Title: Message Title
Kyle Smith commented on an issue
Re: incorrect header check using Ruby 2.1.0
This thread (http://stackoverflow.com/questions/1361892/how-to-decompress-gzip-string-in-ruby) seems to indicate that using inflate on gzip compressed data will result in an 'incorrect header check'. My master is configured with the default mod_deflate config for precise:
IfModule mod_deflate.c
# these are known to be safe with MSIE 6
AddOutputFilterByType DEFLATE text/html text/plain text/xml
# everything else may cause problems with MSIE 6
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE application/x-_javascript_ application/_javascript_ application/ecmascript
AddOutputFilterByType DEFLATE application/rss+xml
/IfModule
Add Comment
Puppet / PUP-1680
incorrect header check using Ruby 2.1.0
I haven't found anything out there so far, so I thought I'd just document it while I'm at it. We ran into some other problem with the changes in net/http Ruby 2.1.0 introduced around the handling of gzip/deflate encoded content. This MIGHT be related. So far we haven't been able to fix it. {code} Error: incorrect header check Error: /Stage[main]/...
This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede)
Jira (PUP-1680) incorrect header check using Ruby 2.1.0
Title: Message Title
Kyle Smith commented on an issue
Re: incorrect header check using Ruby 2.1.0
Just confirmed that the same 4 files threw this error both times.
The files end in .html and .txt. If Puppet is attaching a content-type based on the extension then they're probably the only files being compressed by mod_deflate.
Add Comment
Puppet / PUP-1680
incorrect header check using Ruby 2.1.0
I haven't found anything out there so far, so I thought I'd just document it while I'm at it. We ran into some other problem with the changes in net/http Ruby 2.1.0 introduced around the handling of gzip/deflate encoded content. This MIGHT be related. So far we haven't been able to fix it. {code} Error: incorrect header check Error: /Stage[main]/...
This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede)
--
You received this message because you are subscribed to the Google Groups Puppet Bugs group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at
