#35655 [Fbk->Opn]: Missing end-of-line token

[tomas_matousek at hotmail dot com]

 ID:               35655
 User updated by:  tomas_matousek at hotmail dot com
 Reported By:      tomas_matousek at hotmail dot com
-Status:           Feedback
+Status:           Open
 Bug Type:         Scripting Engine problem
 Operating System: WinXP
 PHP Version:      5.1.1
 New Comment:

The same result with latest snapshot.
The problem is probably here (file zend_language_scanner.l, around line
1750):

                if (unput_semicolon) {
                        unput(';');
                }

I would suggest to replace unputting with 
  yyless(label_len);
which would preserve both ';' and {NEWLINE}.


Previous Comments:
------------------------------------------------------------------------

[2005-12-13 10:58:20] [EMAIL PROTECTED]

Please try using this CVS snapshot:

  http://snaps.php.net/php5.1-latest.tar.gz
 
For Windows:
 
  http://snaps.php.net/win32/php5.1-win32-latest.zip



------------------------------------------------------------------------

[2005-12-13 10:54:25] tomas_matousek at hotmail dot com

Correction:
Expected result of highlighting:
$code = '
<? 
  $x = <<<EOT
some string    
EOT
  $y = 2;
?>'

Actual result:
$code = '
<? 
  $x = <<<EOT
some string    
EOT  $y
 = 2;
?>'

------------------------------------------------------------------------

[2005-12-13 10:52:46] tomas_matousek at hotmail dot com

Description:
------------
PHP lexer omits end-of-line characters following T_END_HEREDOC token.
This also causes incorrect output of highlight_string() function.

Reproduce code:
---------------
$code = '
<? 
  $x = <<<EOT
some string    
EOT
  $y = 2;
?>';

display_tokens(token_get_all($code));
highlight_string($code);



function display_tokens($tokens)
{
  foreach ($tokens as $token)
  { 
    if (is_array($token))
    {
      echo token_name($token[0]),"\n";
      echo $token[0],"
'",htmlentities(addcslashes($token[1],"\n\r\t")),"'\n";
    }
    else
    {
      echo "    '",htmlentities($token),"'\n";
    }  
  }
}

Expected result:
----------------
T_INLINE_HTML
311 '\r\n'
T_OPEN_TAG
367 '<?'
T_WHITESPACE
370 ' \r\n  '
T_VARIABLE
309 '$x'
T_WHITESPACE
370 ' '
    '='
T_WHITESPACE
370 ' '
T_START_HEREDOC
371 '<<<EOT\r\n'
T_STRING
307 'some'
T_ENCAPSED_AND_WHITESPACE
314 ' '
T_STRING
307 'string'
T_ENCAPSED_AND_WHITESPACE
314 '    \r\n'
T_END_HEREDOC
372 'EOT'
T_WHITESPACE
370 '\r\n  '      // '\r\n' is missing in actual result
T_VARIABLE
309 '$y'
T_WHITESPACE
370 ' '
    '='
T_WHITESPACE
370 ' '
T_LNUMBER
305 '2'
    ';'
T_WHITESPACE
370 '\r\n'
T_CLOSE_TAG
369 '?>'


<? 
  $x = <<<EOT
some string    
EOT  $y = 2;
?>

Actual result:
--------------
T_INLINE_HTML
311 '\r\n'
T_OPEN_TAG
367 '<?'
T_WHITESPACE
370 ' \r\n  '
T_VARIABLE
309 '$x'
T_WHITESPACE
370 ' '
    '='
T_WHITESPACE
370 ' '
T_START_HEREDOC
371 '<<<EOT\r\n'
T_STRING
307 'some'
T_ENCAPSED_AND_WHITESPACE
314 ' '
T_STRING
307 'string'
T_ENCAPSED_AND_WHITESPACE
314 '    \r\n'
T_END_HEREDOC
372 'EOT'         // missing '\r\n' 
T_WHITESPACE
370 '  '
T_VARIABLE
309 '$y'
T_WHITESPACE
370 ' '
    '='
T_WHITESPACE
370 ' '
T_LNUMBER
305 '2'
    ';'
T_WHITESPACE
370 '\r\n'
T_CLOSE_TAG
369 '?>'


<? 
  $x = <<<EOT
some string    
EOT  $y = 2;
?>


------------------------------------------------------------------------


-- 
Edit this bug report at http://bugs.php.net/?id=35655&edit=1