Jira (PUP-5728) Change default parsing of manifest files to UTF-8

[Ethan Brown (JIRA)]

Title: Message Title

Ethan Brown created an issue   Puppet / PUP-5728 Change default parsing of manifest files to UTF-8 Issue Type: Bug Assignee: Unassigned Created: 2016/01/20 4:08 PM Priority: Normal Reporter: Ethan Brown As part of PUP-2564 , we changed our language to state that manifest files must be in UTF-8. However, this is not enforced when we actually lex files with the parser, and it causes a host of issues on Windows in particular. https://docs.puppetlabs.com/puppet/latest/reference/lang_summary.html#files As Josh Cooper alluded to, Windows will read the file then treat it as whatever the current codepage is, and this carries a number of issues. For instance, take the manifest user { 'Umlautä': ensure => present, password => 'password' } If running with the codepage 437, the string produced by Puppet::FileSystem.read is incorrect for subsequent usage as ä gets turned into \xC3\xA4: "user { 'Umlaut\xC3\xA4':\n ensure => present,\n password => 'password'\n}\n" So while the user is created, the name is Umlaut├ñ instead of Umlautä do to the way the converted bytes are represented when making the appropriate conversions (i.e calling .encode('utf-8') on the above string produces: "user { 'Umlaut\u251C\u00F1':\n ensure => present,\n password => 'password'\n}\n" If instead the local codepage is set to 65001, which is Unicode... then the behavior is correct. However, this is not something we should expect users to do, and we should treat the incoming file as UTF-8 as that's what our documentation specifies / and what we claim to expect. The prior PUP-2564 ticket mentioned the inability of specifying an encoding during file reading at https://tickets.puppetlabs.com/browse/PUP-2564?focusedCommentId=125526&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-125526. I think that's a problem that can be easily fixed, at least in this narrow use case to start. I spiked a very quick solution to this problem for the sake of discussion. With the change, the parsed string is now represents ä correct as \u00E4: "user { 'Umlaut\u00E4':\n ensure => present,\n password => 'password'\n}\n" I know that it fixes the problem with the manifest above (i.e. you can run any local codepage and the correct user name is used during creation based on the UTF8 file contents). However, there could be some additional fallout, and there are certainly other places where files are read that could take similar tacts. Add Comment   This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.