Issue #19128 has been updated by Pieter van de Bruggen.
JSON is required to be encoded as some form of Unicode. According to the JSON spec: >All Unicode characters may be placed within the quotation marks except for the >characters that must be escaped: quotation mark, reverse solidus, and the >control characters (U+0000 through U+001F). So no escaping is required for Unicode characters. Having said that, it may be necessary to convert the string into a Unicode encoding if it was originally in another encoding (like ASCII 8-bit). Failing to properly handle that introduces the possibility of failure from a number of vectors, but it's most likely to happen in the face of real-world user input – `puppet module build` is one of the most direct paths between user input and the PSON generator. I assert that while there are things that the `build` action could do to mitigate this failure, the PSON library should ultimately be responsible for not breaking in the face of unexpected character encodings. ---------------------------------------- Bug #19128: "puppet module build" doesn't escape PSON correctly https://projects.puppetlabs.com/issues/19128#change-83137 Author: Pierre Carrier Status: Investigating Priority: High Assignee: Ryan Coleman Category: module tool Target version: 3.1.x Affected Puppet version: 3.1.0 Keywords: Branch: Erik Dalén's modules are incompatible with ruby 1.9.3p327 and puppet 3.1.0 because he's Swedish. As soon as I specify `[main] modulepath = /Users/pierre/repos/spotify-puppet/modules in ~/.puppet/puppet.conf`, Puppet gets unhappy: % puppet help /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/external/pson/pure/parser.rb:154:in `rescue in parse_string': Caught Encoding::CompatibilityError: incompatible encoding regexp match (ASCII-8BIT regexp with UTF-8 string) (PSON::GeneratorError) from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/external/pson/pure/parser.rb:133:in `parse_string' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/external/pson/pure/parser.rb:229:in `parse_object' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/external/pson/pure/parser.rb:98:in `parse' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/external/pson/common.rb:133:in `parse' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/module.rb:55:in `has_metadata?' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/module.rb:45:in `initialize' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/node/environment.rb:141:in `new' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/node/environment.rb:141:in `block (2 levels) in <class:Environment>' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/node/environment.rb:139:in `collect' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/node/environment.rb:139:in `block in <class:Environment>' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/util/cacher.rb:60:in `block in cached_value' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/util/cacher.rb:58:in `cached_value' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/util/cacher.rb:31:in `block in cached_attr' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/node/environment.rb:96:in `each_plugin_directory' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/util/command_line.rb:124:in `run' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/util/command_line.rb:86:in `execute' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/bin/puppet:4:in `<top (required)>' from /Users/pierre/.gem/ruby/1.9.3/bin/puppet:23:in `load' from /Users/pierre/.gem/ruby/1.9.3/bin/puppet:23:in `<main>' zsh: exit 1 puppet help/Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/external/pson/pure/parser.rb:154:in `rescue in parse_string': Caught Encoding::CompatibilityError: incompatible encoding regexp match (ASCII-8BIT regexp with UTF-8 string) (PSON::GeneratorError) from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/external/pson/pure/parser.rb:133:in `parse_string' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/external/pson/pure/parser.rb:229:in `parse_object' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/external/pson/pure/parser.rb:98:in `parse' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/external/pson/common.rb:133:in `parse' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/module.rb:55:in `has_metadata?' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/module.rb:45:in `initialize' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/node/environment.rb:141:in `new' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/node/environment.rb:141:in `block (2 levels) in <class:Environment>' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/node/environment.rb:139:in `collect' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/node/environment.rb:139:in `block in <class:Environment>' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/util/cacher.rb:60:in `block in cached_value' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/util/cacher.rb:58:in `cached_value' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/util/cacher.rb:31:in `block in cached_attr' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/node/environment.rb:96:in `each_plugin_directory' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/util/command_line.rb:124:in `run' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/lib/puppet/util/command_line.rb:86:in `execute' from /Users/pierre/.rubies/1.9.3-p327/lib/ruby/gems/1.9.1/gems/puppet-3.1.0/bin/puppet:4:in `<top (required)>' from /Users/pierre/.gem/ruby/1.9.3/bin/puppet:23:in `load' from /Users/pierre/.gem/ruby/1.9.3/bin/puppet:23:in `<main>' zsh: exit 1 puppet help I decided not to investigate the PSON code as it's past bedtime. But from my understanding, the PSON parser.rb expects `\u????` sequences for non-ASCII characters and rejects inline Unicode characters. The PSON generator.rb seems supposed to take care of the escaping, so his `Modulefile`s are OK. I reproduced the issue by regenerating `metadata.json` with my versions, ruby 1.9.3p327 and puppet 3.1.0 (GNU grep needed for this): puppet-puppetdbquery % puppet module buildNotice: Building /Users/pierre/repos/puppet-puppetdbquery for release Module built: /Users/pierre/repos/puppet-puppetdbquery/pkg/dalen-puppetdbquery-0.1.0.tar.gz % tar xfO pkg/dalen-puppetdbquery-0.1.0.tar.gz dalen-puppetdbquery-0.1.0/metadata.json|grep -P '[\x80-\xff]' "author": "Erik Dalén <[email protected]>", -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/puppet-bugs?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
