Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Ethan Brown updated an issue Puppet / PUP-5728 Change default parsing of manifest files to UTF-8 Change By: Ethan Brown Labels: i18n utf-8 Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Ethan Brown updated an issue Puppet / PUP-5728 Change default parsing of manifest files to UTF-8 Change By: Ethan Brown Release Notes Summary: As part of Puppet 4, Puppet declared that UTF-8 was the only valid encoding for manifest files. However, while this behavior was correct on non-Windows systems, because Puppet did not explicitly specify the encoding when reading manifests, the behavior on Windows was incorrect. On Windows, Ruby continued to load files from disk, then convert them to whatever the current local codepage was defined as (often IBM437 or 1252). Depending on whether the Unicode characters in the manifest contained bytes that could be represented in the current codepage or not, this could lead to either a crash while attempting to treat the bytes as characters in the codepage, or more typically, corruption of the intended strings when loading internally. This corruption would result in resources being created that didn't match what was specified in the manifest. In addition to addressing the loading of manifest files, similar changes were made to the loading of resource templates, Ruby based Puppet module code (such as custom functions), puppet apply, puppet lookup, the epp and parser faces. Release Notes: Bug Fix Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Josh Cooper updated an issue Puppet / PUP-5728 Change default parsing of manifest files to UTF-8 Change By: Josh Cooper Fix Version/s: PUP 4.3.3 Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title
Ryan Gard commented on PUP-5728
Re: Change default parsing of manifest files to UTF-8
Functional Review
Verified as fixed with build 6522aa18d8338b9ffdc613ea7e750a633d4ec6d5.
Environment
Agent: Windows Server 2012 R2 (WIN_SUT)
Test Case 1 Reproduction Steps
Install Puppet Agent on WIN_SUT.
Create the ERB template "C:\template.erb" (UTF-8 Encoding) on the WIN_SUT with the following content:
尻屌屛屗尐少
<%= @text %>
漕滴滸漛漣
Create the manifest "C:\test.pp" (UTF-8 Encoding) on the WIN_SUT with the following content:
$text='ᅪᅫᅇᅽᄟ'
file { 'c:\漕滴滸漛漣':
ensure => file,
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Ryan Gard commented on PUP-5728 Re: Change default parsing of manifest files to UTF-8 Functional Review Verified as fixed with build 6522aa18d8338b9ffdc613ea7e750a633d4ec6d5. Environment Agent: Windows Server 2012 R2 (WIN_SUT) Agent: CentOS 7 x64 (LINUX_SUT) Test Case 1 Reproduction Steps Install Puppet Agent on WIN_SUT. Create the EPP template "C:\template.epp" (UTF-8 Encoding) on the WIN_SUT with the following content: 尻屌屛屗尐少 <%= $text %> 漕滴滸漛漣 Install ConEmu. (This enables displaying Unicode content without changing codepage.) Open a ConEmu console session. Verify that codepage is 437: chcp
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title
Ryan Gard commented on PUP-5728
Re: Change default parsing of manifest files to UTF-8
Functional Review
Verified as fixed with build 6522aa18d8338b9ffdc613ea7e750a633d4ec6d5.
Environment
Agent: Windows Server 2012 R2 (WIN_SUT)
Test Case 1 Reproduction Steps
Install Puppet Agent on WIN_SUT.
Create the EPP template "C:\template.epp" (UTF-8 Encoding) on the WIN_SUT with the following content:
尻屌屛屗尐少
<%= $text %>
漕滴滸漛漣
Create the manifest "C:\test.pp" (UTF-8 Encoding) on the WIN_SUT with the following content:
$text='ᅪᅫᅇᅽᄟ'
file { 'c:\漕滴滸漛漣':
ensure => file,
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title
Ryan Gard commented on PUP-5728
Re: Change default parsing of manifest files to UTF-8
Functional Review
Verified as fixed with build 6522aa18d8338b9ffdc613ea7e750a633d4ec6d5.
Environment
Agent: Windows Server 2012 R2 (WIN_SUT)
Test Case 1 Reproduction Steps
Install Puppet Agent on WIN_SUT.
Create the manifest "C:\test.pp" (UTF-8 Encoding) on the WIN_SUT with the following content:
file { "c:\\unicode_\u5BF9\u5BFA\u5BFB\u5BFD\u5BFE\u5BFF\u5BF8":
ensure => file,
content => "\u5BF9\u5BFA\u5BFB\u5BFD\u5BFE\u5BFF\u5BF8",
}
file { 'c:\漕滴滸漛漣':
ensure => file,
content => '漕滴滸漛漣',
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Ethan Brown commented on PUP-5728 Re: Change default parsing of manifest files to UTF-8 Ah yeah, I understand now what you were trying to do. Unfortunately that also works around the Ruby behavior the PR addresses. (Ruby previously converted the manifest to the local codepage when reading the file - but if the local codepage is set to UTF8 (65001), then there are no conversion problems) I can tell you that if you use ConEmu, it should display Unicode characters correctly, even when the local codepage is set to 437 and not 65001. Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Ryan Gard commented on PUP-5728 Re: Change default parsing of manifest files to UTF-8 Ethan Brown The codepage was set to 65001 so that the console would properly display Unicode characters for verification reasons. I'll retest with 437 today just to make sure. Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Ethan Brown commented on PUP-5728 Re: Change default parsing of manifest files to UTF-8 Ryan Gard - I somehow missed earlier that you had changed the codepage to 65001 during FR. Per the original ticket description, these behavioral changes should exist even without the code page being set like that. In other words, even with a codepage of 437 or 1252 set, manifests should always be interpreted as UTF-8 without any additional / out of the ordinary config changes by an end user, like forcefully setting chcp 65001. In fact, Puppet previously behaved properly parsing / applying manifests when that codepage was set. Sorry I missed this earlier, but we should probably double check this with 437 just to be on the safe side. The unit tests already ensure that 65001 isn't set (so that we don't get false positive test passes) - https://github.com/puppetlabs/puppet/blob/c384ddacdc24167f6395cc4c29e1dd28a7ab4018/spec/unit/pops/parser/lexer2_spec.rb#L755-L757 Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Larissa Lane updated an issue Puppet / PUP-5728 Change default parsing of manifest files to UTF-8 Change By: Larissa Lane Labels: i18n Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title
Ryan Gard commented on PUP-5728
Re: Change default parsing of manifest files to UTF-8
Functional Review
Verified as fixed with build 6522aa18d8338b9ffdc613ea7e750a633d4ec6d5.
Environment
Agent: Windows Server 2012 R2 (WIN_SUT) Agent: CentOS 7 x64 (LINUX_SUT)
Test Case 1 Reproduction Steps
Install Puppet Agent on WIN_SUT.
Create the ERB template "C:\template.erb" (UTF-8 Encoding) on the WIN_SUT with the following content:
尻屌屛屗尐少
<%= @text %>
漕滴滸漛漣
Create the manifest "C:\test.pp" (UTF-8 Encoding) on the WIN_SUT with the following content:
$text='ᅪᅫᅇᅽᄟ'
file { 'c:\漕滴滸漛漣':
ensure => file,
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Ryan Gard commented on PUP-5728 Re: Change default parsing of manifest files to UTF-8 Functional Review Verified as fixed with build 6522aa18d8338b9ffdc613ea7e750a633d4ec6d5. Environment Agent: Windows Server 2012 R2 (WIN_SUT) Agent: CentOS 7 x64 (LINUX_SUT) Test Case 1 Reproduction Steps Install Puppet Agent on WIN_SUT. Create the EPP template "C:\template.epp" (UTF-8 Encoding) on the WIN_SUT with the following content: 尻屌屛屗尐少 <%= $text %> 漕滴滸漛漣 Change console font to "Lucidia Console". (Puppet will crash otherwise. PUP-5717 ) Change codepage: chcp 65001
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title
Ryan Gard commented on PUP-5728
Re: Change default parsing of manifest files to UTF-8
Functional Review
Verified as fixed with build 6522aa18d8338b9ffdc613ea7e750a633d4ec6d5.
Environment
Agent: Windows Server 2012 R2 (WIN_SUT) Agent: CentOS 7 x64 (LINUX_SUT)
Test Case 1 Reproduction Steps
Install Puppet Agent on WIN_SUT.
Create the EPP template "C:\template.epp" (UTF-8 Encoding) on the WIN_SUT with the following content:
尻屌屛屗尐少
<%= $text %>
漕滴滸漛漣
Create the manifest "C:\test.pp" (UTF-8 Encoding) on the WIN_SUT with the following content:
$text='ᅪᅫᅇᅽᄟ'
file { 'c:\漕滴滸漛漣':
ensure => file,
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title
Ryan Gard commented on PUP-5728
Re: Change default parsing of manifest files to UTF-8
Functional Review
Verified as fixed with build 6522aa18d8338b9ffdc613ea7e750a633d4ec6d5.
Environment
Agent: Windows Server 2012 R2 (WIN_SUT) Agent: CentOS 7 x64 (LINUX_SUT)
Test Case 1 Reproduction Steps
Install Puppet Agent on WIN_SUT.
Create the manifest "C:\test.pp" (UTF-8 Encoding) on the WIN_SUT with the following content:
file { "c:\\unicode_\u5BF9\u5BFA\u5BFB\u5BFD\u5BFE\u5BFF\u5BF8":
ensure => file,
content => "\u5BF9\u5BFA\u5BFB\u5BFD\u5BFE\u5BFF\u5BF8",
}
file { 'c:\漕滴滸漛漣':
ensure => file,
content => '漕滴滸漛漣',
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Ryan Gard commented on PUP-5728 Re: Change default parsing of manifest files to UTF-8 After talking with Ethan Brown I am going to defer that last bit of FR concerning "puppet lookup" to a later sprint. See PUP-5851 for more details. Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Ryan Gard assigned an issue to Ethan Brown Puppet / PUP-5728 Change default parsing of manifest files to UTF-8 Change By: Ryan Gard Resolution: Fixed Assignee: Ryan Gard Ethan Brown Status: Ready for Test Resolved Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Glenn Sarti commented on PUP-5728 Re: Change default parsing of manifest files to UTF-8 Artifacts are available at http://builds.puppetlabs.lan/puppet-agent/6522aa18d8338b9ffdc613ea7e750a633d4ec6d5/artifacts/windows/ Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Glenn Sarti assigned an issue to qa-team Puppet / PUP-5728 Change default parsing of manifest files to UTF-8 Change By: Glenn Sarti Assignee: qa -team Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Glenn Sarti assigned an issue to qa Puppet / PUP-5728 Change default parsing of manifest files to UTF-8 Change By: Glenn Sarti Status: Ready for CI Test Assignee: Glenn Sarti qa Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Ryan Gard assigned an issue to Ryan Gard Puppet / PUP-5728 Change default parsing of manifest files to UTF-8 Change By: Ryan Gard Assignee: qa-team Ryan Gard Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Glenn Sarti commented on PUP-5728 Re: Change default parsing of manifest files to UTF-8 Merged into stable at https://github.com/puppetlabs/puppet/commit/83b1e6e64ffc9aa3ed2b710c06a2e4ee72c6897d Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Ethan Brown assigned an issue to Glenn Sarti Puppet / PUP-5728 Change default parsing of manifest files to UTF-8 Change By: Ethan Brown Assignee: Glenn Sarti Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Steve Barlow updated an issue Puppet / PUP-5728 Change default parsing of manifest files to UTF-8 Change By: Steve Barlow Scope Change Reason: Discovered while testing PUP-5538 and fixed worked on while in the area Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Steve Barlow updated an issue Puppet / PUP-5728 Change default parsing of manifest files to UTF-8 Change By: Steve Barlow Scope Change Category: Adopted Scope Change Reason: Discovered while testing PUP-5538 and fixed while in the area Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Steve Barlow updated an issue Puppet / PUP-5728 Change default parsing of manifest files to UTF-8 Change By: Steve Barlow Sprint: Windows 2016-01-27 , Windows 2016-02-10 Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Ryan Gard updated an issue Puppet / PUP-5728 Change default parsing of manifest files to UTF-8 Change By: Ryan Gard QA Risk Assessment Reason: Developer requested review and felt that unit tests give decent regression coverage. QA Risk Assessment: Medium QA Status: Reviewed QA Contact: Ryan Gard Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Ethan Brown updated an issue Puppet / PUP-5728 Change default parsing of manifest files to UTF-8 Change By: Ethan Brown Fix Version/s: PUP 4.4.0 Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Ethan Brown updated an issue Puppet / PUP-5728 Change default parsing of manifest files to UTF-8 Change By: Ethan Brown Fix Version/s: PUP 4.3.3 Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Ethan Brown updated an issue Puppet / PUP-5728 Change default parsing of manifest files to UTF-8 Change By: Ethan Brown Sprint: Windows 2016-01-27 Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Ethan Brown assigned an issue to Ethan Brown Puppet / PUP-5728 Change default parsing of manifest files to UTF-8 Change By: Ethan Brown Assignee: Ethan Brown Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title
Ethan Brown updated an issue
Puppet / PUP-5728
Change default parsing of manifest files to UTF-8
Change By:
Ethan Brown
As part of PUP-2564, we changed our language to state that manifest files must be in UTF-8. However, this is not enforced when we actually lex files with the parser, and it causes a host of issues on Windows in particular.https://docs.puppetlabs.com/puppet/latest/reference/lang_summary.html#filesAs [~josh] alluded to, Windows will read the file then treat it as whatever the current codepage is, and this carries a number of issues.For instance, take the manifest {code}user { 'Umlautä': ensure => present, password => 'password'}{code}If running with the codepage 437, the string produced by {{Puppet::FileSystem.read}} is incorrect for subsequent usage as {{ä}} gets turned into {{\xC3\xA4}}:{code}"user { 'Umlaut\xC3\xA4':\n ensure => present,\n password => 'password'\n}\n"{code}So while the user is created, the name is {{Umlaut├ñ}} instead of {{Umlautä}} do due to the way the converted bytes are represented when making the appropriate conversions (i.e calling {{.encode('utf-8')}} on the above string produces:{code}"user { 'Umlaut\u251C\u00F1':\n ensure => present,\n password => 'password'\n}\n"{code}If instead the local codepage is set to {{65001}}, which is Unicode... then the behavior is correct. However, this is not something we should expect users to do, and we should treat the incoming file as UTF-8 as that's what our documentation specifies / and what we claim to expect.The prior PUP-2564 ticket mentioned the inability of specifying an encoding during file reading at https://tickets.puppetlabs.com/browse/PUP-2564?focusedCommentId=125526=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-125526. I think that's a problem that can be easily fixed, at least in this narrow use case to start.I spiked a very quick solution to this problem for the sake of discussion. With the change, the parsed string is now represents {{ä}} correct as {{\u00E4}}:{code}"user { 'Umlaut\u00E4':\n ensure => present,\n password => 'password'\n}\n"{code}I know that it fixes the problem with the manifest above (i.e. you can run any local codepage and the correct user name is used during creation based on the UTF8 file contents). However, there could be some additional fallout, and there are certainly other places where files are read that could take similar tacts.
Add Comment
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title Ethan Brown commented on PUP-5728 Re: Change default parsing of manifest files to UTF-8 Also worth noting is that we could use a more robust detection method for file encodings (like the one used in charlock_holmes at https://github.com/brianmario/charlock_holmes/blob/master/ext/charlock_holmes/encoding_detector.c). However, given we've specified that manifests are supposed to be in UTF-8, I think that we keep it simple as the PR attempts to. Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5728) Change default parsing of manifest files to UTF-8
Title: Message Title
Ethan Brown created an issue
Puppet / PUP-5728
Change default parsing of manifest files to UTF-8
Issue Type:
Bug
Assignee:
Unassigned
Created:
2016/01/20 4:08 PM
Priority:
Normal
Reporter:
Ethan Brown
As part of
PUP-2564
, we changed our language to state that manifest files must be in UTF-8. However, this is not enforced when we actually lex files with the parser, and it causes a host of issues on Windows in particular.
https://docs.puppetlabs.com/puppet/latest/reference/lang_summary.html#files
As Josh Cooper alluded to, Windows will read the file then treat it as whatever the current codepage is, and this carries a number of issues.
For instance, take the manifest
user { 'Umlautä':
ensure => present,
password => 'password'
