Jira (PUP-5584) Cached catalogs are loaded using the agent's string locale which can result in corrupted data

[Josh Cooper (JIRA)]

Title: Message Title

Josh Cooper commented on PUP-5584   Re: Cached catalogs are loaded using the agent's string locale which can result in corrupted data Couple of notes about what Branan Riley and I talked about leading up to merging this. We call Puppet::Indirector::JSON#to_json to save the catalog to the cache. The method converts the catalog to a Hash, and calls into the PSON (pure json) library that we vendor. PSON serializes the nested structure, and for each string, e.g. hash value, calls String#to_json. The PSON library's utf8_to_pson method encodes the string as PSON, and returns it as an ASCII-8BIT encoded string. It appears the latest version of the upstream pson library returns a UTF-8 encoded string instead. We decided to not try to update our vendored PSON library right now. After serializing the catalog to pson, we force the encoding to ASCII-8BIT so that ruby will not transcode the contents when writing the pson-encoded string to the file (based on the default external encoding). This step is not strictly necessary, because due to the above issue, the pson-encoded catalog is already ASCII-8BIT encoded. However, that could change in the future, if/when we update our vendored pson. When reading in the cached catalog, ruby will force encode the string contents based on the current external encoding. On Windows, this depends on the current code page, e.g. CP437. On *nix, it is typically UTF-8 though doesn't have to be. In either case, we need to override ruby's force encoding so that the catalog we read in has is tagged with an encoding that matches what it started out in, i.e. ASCII-8BIT. Prior to this PR, when reading the cached catalog, we were feeding the Windows-specific encoded string into PSON.from_json. If given something other than ASCII-8BIT, then from_json calls source.encode(::Encoding::UTF_8) which causes the BOM corruption described in the ticket. In the future, we may want to patch our vendored pson library. At that point, we should force_encoding the catalog read from disk to UTF-8. Additional info: replace_file uses Puppet::FileSystem::Uniquefile to open the file for writing, which defaults to text mode. On Windows, this will cause each \n to be written out as \r\n. However, to_pson escapes newlines, so the pson-encoded string contains the byte sequence [backslash, r, backslash, n]. As a result, no CRNL translation takes place on Windows, and we don't need to worry about text vs binary modes. The PSON library assumes utf8_to_json receives a UTF-8 encoded string, but that is not always the case. It appears that any data read from puppet.conf has an encoding based on the default external encoding: On Windows: Notice: Encoding US-ASCII: service_provider Notice: Encoding UTF-8: windows Notice: Encoding UTF-8: clientcert Notice: Encoding Windows-1252: agent1 Notice: Encoding UTF-8: clientversion Notice: Encoding UTF-8: 4.3.2 On *nix with an alternate encoding: $ env LC_CTYPE=de_DE.ISO8859-1 bundle exec puppet agent -t --server arcturus ... Notice: Encoding UTF-8: /var/root Notice: Encoding US-ASCII: service_provider Notice: Encoding UTF-8: launchd Notice: Encoding UTF-8: clientcert Notice: Encoding ISO-8859-1: agent2 We should file a separate ticket to ensure data read from settings files is converted to UTF-8 when read in. Add Comment   This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.