This is a modification of the Nick/Jesse/Matt patch, retaining their tests and the analysis of the problem but reversing the implementation direction of the solution.
Rather than trying to make the already somewhat brittle slurpstring smarter, which requires telling it what following strings will be accepted by the caller with a zero-width-lookahead negation of the regular expression used to extract a variable name, this patch keeps that responsibility in the caller where it belongs. The caller (tokenize_interpolated_string) now checks to see if it got a variable name _before_ emitting a variable token; if it got one, it proceeds normally, but if it didn't it simply tries again from that point in the string (accumulating the false match as a prefix). This change actually simplifies the logic of tokenize_interpolated_string somewhat. Signed-off-by: Markus Roberts <[email protected]> --- lib/puppet/parser/lexer.rb | 16 +++++++++------- spec/unit/parser/lexer_spec.rb | 8 +++++++- 2 files changed, 16 insertions(+), 8 deletions(-) diff --git a/lib/puppet/parser/lexer.rb b/lib/puppet/parser/lexer.rb index 6a9f1cf..1e10ff9 100644 --- a/lib/puppet/parser/lexer.rb +++ b/lib/puppet/parser/lexer.rb @@ -540,15 +540,17 @@ class Puppet::Parser::Lexer [ str[0..-2],str[-1,1] ] end - def tokenize_interpolated_string(token_type) + def tokenize_interpolated_string(token_type,preamble='') value,terminator = slurpstring('"$') - token_queue << [TOKENS[token_type[terminator]],value] - while terminator == '$' and not @scanner.scan(/\{/) - token_queue << [TOKENS[:VARIABLE],@scanner.scan(%r{(\w*::)*\w+|[0-9]})] - value,terminator = slurpstring('"$') - token_queue << [TOKENS[DQ_continuation_token_types[terminator]],value] + token_queue << [TOKENS[token_type[terminator]],preamble+value] + if terminator != '$' or @scanner.scan(/\{/) + token_queue.shift + elsif var_name = @scanner.scan(%r{(\w*::)*\w+|[0-9]}) + token_queue << [TOKENS[:VARIABLE],var_name] + tokenize_interpolated_string(DQ_continuation_token_types) + else + tokenize_interpolated_string(token_type,token_queue.pop.last + terminator) end - token_queue.shift end # just parse a string, not a whole file diff --git a/spec/unit/parser/lexer_spec.rb b/spec/unit/parser/lexer_spec.rb index 81e76a3..d3d2a0a 100755 --- a/spec/unit/parser/lexer_spec.rb +++ b/spec/unit/parser/lexer_spec.rb @@ -426,7 +426,13 @@ describe Puppet::Parser::Lexer,"when lexing strings" do %q{a hardest "scanner \"test\""} => [[:NAME,"a"],[:NAME,"hardest"],[:STRING,'scanner "test"']], %Q{a hardestest "scanner \\"test\\"\n"} => [[:NAME,"a"],[:NAME,"hardestest"],[:STRING,%Q{scanner "test"\n}]], %q{function("call")} => [[:NAME,"function"],[:LPAREN,"("],[:STRING,'call'],[:RPAREN,")"]], - %q["string with ${(3+5)/4} nested math."] => [[:DQPRE,"string with "],:LPAREN,[:NAME,"3"],:PLUS,[:NAME,"5"],:RPAREN,:DIV,[:NAME,"4"],[:DQPOST," nested math."]] + %q["string with ${(3+5)/4} nested math."] => [[:DQPRE,"string with "],:LPAREN,[:NAME,"3"],:PLUS,[:NAME,"5"],:RPAREN,:DIV,[:NAME,"4"],[:DQPOST," nested math."]], + %q["$$$$"] => [[:STRING,"$$$$"]], + %q["$variable"] => [[:DQPRE,""],[:VARIABLE,"variable"],[:DQPOST,""]], + %q["$var$other"] => [[:DQPRE,""],[:VARIABLE,"var"],[:DQMID,""],[:VARIABLE,"other"],[:DQPOST,""]], + %q["foo$bar$"] => [[:DQPRE,"foo"],[:VARIABLE,"bar"],[:DQPOST,"$"]], + %q["foo$$bar"] => [[:DQPRE,"foo$"],[:VARIABLE,"bar"],[:DQPOST,""]], + %q[""] => [[:STRING,""]], }.each { |src,expected_result| it "should handle #{src} correctly" do tokens_scanned_from(src).should be_like(*expected_result) -- 1.6.4 -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.
