Jira (PUP-7251) gzip decompression manges utf-8 content in catalog

[Moses Mendoza (JIRA)]

Title: Message Title

Moses Mendoza created an issue   Puppet / PUP-7251 gzip decompression manges utf-8 content in catalog Issue Type: Bug Assignee: Unassigned Created: 2017/02/23 10:29 AM Priority: Normal Reporter: Moses Mendoza It appears a regression in UTF-8 handling by Puppet was introduced by https://github.com/puppetlabs/puppetserver/commit/b7d8ff4ab2603ef94f2f09ebcf42e535b69033a3. This was discovered in investigating a local failure of the agent 1.9 series against https://github.com/puppetlabs/puppet/blob/master/acceptance/tests/resource/file/ticket_6448_file_with_utf8_source.rb against puppet server 2.7.2.master-0.1SNAPSHOT.2017.02.22T1106. We are not seeing this failure in CI presumably because CI is pinned to a released version of puppet server without this change. The catalog returned by the server in this test contains a UTF-8 named file resource, "\u9759\u7864" Prior to the change, the catalog received by the agent was uncompressed and contained: [5] pry(#<Puppet::Resource::Catalog::Rest>)> response.body => "{\"tags\":[\"settings\"],\"name\":\"cpq55y0vuphp44u.delivery.puppetlabs.net\",\"version\":1487871618,\"code_id\":null,\"catalog_uuid\":\"cf89d5fd-ebdf-426a-832b-68812066c165\",\"catalog_format\":1,\"environment\":\"ticket_6448_file_with_utf8_source_j9w3rhmk\",\"resources\":[{\"type\":\"Stage\",\"title\":\"main\",\"tags\":[\"stage\"],\"exported\":false,\"parameters\":{\"name\":\"main\"}},{\"type\":\"Class\",\"title\":\"Settings\",\"tags\":[\"class\",\"settings\"],\"exported\":false},{\"type\":\"Class\",\"title\":\"main\",\"tags\":[\"class\"],\"exported\":false,\"parameters\":{\"name\":\"main\"}},{\"type\":\"File\",\"title\":\"/tmp/ticket_6448_file_with_utf8_source_j9w3rhmk.7EeVuV/\xEF\xBD\xB2\xEF\xBD\xA7\xE3\x83\x95\xE3\x83\xAB\",\"tags\":[\"file\",\"class\"],\"file\":\"/etc/puppetlabs/code/environments/ticket_6448_file_with_utf8_source_j9w3rhmk/manifests/site.pp\",\"line\":2,\"exported\":false,\"parameters\":{\"ensure\":\"present\",\"source\":\"puppet:///modules/utf8_file_module/\xE9\x9D\x99\xE7\x9A\x84\"}}],\"edges\":[{\"source\":\"Stage[main]\",\"target\":\"Class[Settings]\"},{\"source\":\"Stage[main]\",\"target\":\"Class[main]\"},{\"source\":\"Class[main]\",\"target\":\"File[/tmp/ticket_6448_file_with_utf8_source_j9w3rhmk.7EeVuV/\xEF\xBD\xB2\xEF\xBD\xA7\xE3\x83\x95\xE3\x83\xAB]\"}],\"classes\":[\"settings\"]}" It thus did not go through gzip decompression, here: https://github.com/puppetlabs/puppet/blob/master/lib/puppet/network/http/compression.rb#L23 After the change, the catalog requires decompression - and the decompressed result is mangled UTF-8: [4] pry(#<Puppet::Resource::Catalog::Rest>)> Zlib::GzipReader.new(StringIO.new(response.body)).read => "{\"tags\":[\"settings\"],\"name\":\"cpq55y0vuphp44u.delivery.puppetlabs.net\",\"version\":1487871046,\"code_id\":null,\"catalog_uuid\":\"413cbbbb-9035-4ffd-84bd-d792a7cf7976\",\"catalog_format\":1,\"environment\":\"ticket_6448_file_with_utf8_source_j9w3rhmk\",\"resources\":[{\"type\":\"Stage\",\"title\":\"main\",\"tags\":[\"stage\"],\"exported\":false,\"parameters\":{\"name\":\"main\"}},{\"type\":\"Class\",\"title\":\"Settings\",\"tags\":[\"class\",\"settings\"],\"exported\":false},{\"type\":\"Class\",\"title\":\"main\",\"tags\":[\"class\"],\"exported\":false,\"parameters\":{\"name\":\"main\"}},{\"type\":\"File\",\"title\":\"/tmp/ticket_6448_file_with_utf8_source_j9w3rhmk.7EeVuV/ｲｧã\x83\x95ã\x83«\",\"tags\":[\"file\",\"class\"],\"file\":\"/etc/puppetlabs/code/environments/ticket_6448_file_with_utf8_source_j9w3rhmk/manifests/site.pp\",\"line\":2,\"exported\":false,\"parameters\":{\"ensure\":\"present\",\"source\":\"puppet:///modules/utf8_file_module/é\x9D\x99ç\x9A\x84\"}}],\"edges\":[{\"source\":\"Stage[main]\",\"target\":\"Class[Settings]\"},{\"source\":\"Stage[main]\",\"target\":\"Class[main]\"},{\"source\":\"Class[main]\",\"target\":\"File[/tmp/ticket_6448_file_with_utf8_source_j9w3rhmk.7EeVuV/ｲｧã\x83\x95ã\x83«]\"}],\"classes\":[\"settings\"]}" This is because ruby's Zlib::GzipReader will default to decompressing as Encoding.default_external - in our environment this is not always UTF-8. The solution is to specify UTF-8 encoding as part of the decompression, ie at: https://github.com/puppetlabs/puppet/blob/master/lib/puppet/network/http/compression.rb#L23: Zlib::GzipReader.new(StringIO.new(response.body), :encoding => Encoding::UTF_8).read Note - we also rely on Zlib::Inflate but this defaults to decompressing as ASCII-8BIT rather than default-external, which may be appropriate for our purposes. Add Comment   This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.