From: master dot jexus at gmail dot com
Operating system: Windows XP SP3
PHP version: 5.3.0alpha3
PHP Bug Type: Unknown/Other Function
Bug description: tokenizer misses last single-line comment
Description:
------------
When using the tokenizer to lex given text, the output seems to miss
the last token, if it was a single line comment.
It only seems to occur if there isn't a newline behind the comment
lexeme.
Note the last entries in the arrays.
Reproduce code:
---------------
<?php
print_r(token_get_all(file_get_contents(__FILE__)));
// test
$var = 5;
// test
Expected result:
----------------
Array
(
[0] => Array
(
[0] => 367
[1] => 1
)
[1] => Array
(
[0] => 307
[1] => print_r
[2] => 2
)
[2] => (
[3] => Array
(
[0] => 307
[1] => token_get_all
[2] => 2
)
[4] => (
[5] => Array
(
[0] => 307
[1] => file_get_contents
[2] => 2
)
[6] => (
[7] => Array
(
[0] => 364
[1] => __FILE__
[2] => 2
)
[8] => )
[9] => )
[10] => )
[11] => ;
[12] => Array
(
[0] => 370
[1] =>
[2] => 2
)
[13] => Array
(
[0] => 365
[1] => // test
[2] => 4
)
[14] => Array
(
[0] => 309
[1] => $var
[2] => 5
)
[15] => Array
(
[0] => 370
[1] =>
[2] => 5
)
[16] => =
[17] => Array
(
[0] => 370
[1] =>
[2] => 5
)
[18] => Array
(
[0] => 305
[1] => 5
[2] => 5
)
[19] => ;
[20] => Array
(
[0] => 370
[1] =>
[2] => 5
)
[21] => Array
(
[0] => 365
[1] => // test
[2] => 6
)
)
Actual result:
--------------
Array
(
[0] => Array
(
[0] => 368
[1] => 1
)
[1] => Array
(
[0] => 307
[1] => print_r
[2] => 2
)
[2] => (
[3] => Array
(
[0] => 307
[1] => token_get_all
[2] => 2
)
[4] => (
[5] => Array
(
[0] => 307
[1] => file_get_contents
[2] => 2
)
[6] => (
[7] => Array
(
[0] => 365
[1] => __FILE__
[2] => 2
)
[8] => )
[9] => )
[10] => )
[11] => ;
[12] => Array
(
[0] => 371
[1] =>
[2] => 2
)
[13] => Array
(
[0] => 366
[1] => // test
[2] => 4
)
[14] => Array
(
[0] => 309
[1] => $var
[2] => 5
)
[15] => Array
(
[0] => 371
[1] =>
[2] => 5
)
[16] => =
[17] => Array
(
[0] => 371
[1] =>
[2] => 5
)
[18] => Array
(
[0] => 305
[1] => 5
[2] => 5
)
[19] => ;
[20] => Array
(
[0] => 371
[1] =>
[2] => 5
)
)
--
Edit bug report at http://bugs.php.net/?id=46817&edit=1
--
Try a CVS snapshot (PHP 5.2):
http://bugs.php.net/fix.php?id=46817&r=trysnapshot52
Try a CVS snapshot (PHP 5.3):
http://bugs.php.net/fix.php?id=46817&r=trysnapshot53
Try a CVS snapshot (PHP 6.0):
http://bugs.php.net/fix.php?id=46817&r=trysnapshot60
Fixed in CVS:
http://bugs.php.net/fix.php?id=46817&r=fixedcvs
Fixed in CVS and need be documented:
http://bugs.php.net/fix.php?id=46817&r=needdocs
Fixed in release:
http://bugs.php.net/fix.php?id=46817&r=alreadyfixed
Need backtrace:
http://bugs.php.net/fix.php?id=46817&r=needtrace
Need Reproduce Script:
http://bugs.php.net/fix.php?id=46817&r=needscript
Try newer version:
http://bugs.php.net/fix.php?id=46817&r=oldversion
Not developer issue:
http://bugs.php.net/fix.php?id=46817&r=support
Expected behavior:
http://bugs.php.net/fix.php?id=46817&r=notwrong
Not enough info:
http://bugs.php.net/fix.php?id=46817&r=notenoughinfo
Submitted twice:
http://bugs.php.net/fix.php?id=46817&r=submittedtwice
register_globals:
http://bugs.php.net/fix.php?id=46817&r=globals
PHP 4 support discontinued: http://bugs.php.net/fix.php?id=46817&r=php4
Daylight Savings: http://bugs.php.net/fix.php?id=46817&r=dst
IIS Stability:
http://bugs.php.net/fix.php?id=46817&r=isapi
Install GNU Sed:
http://bugs.php.net/fix.php?id=46817&r=gnused
Floating point limitations:
http://bugs.php.net/fix.php?id=46817&r=float
No Zend Extensions:
http://bugs.php.net/fix.php?id=46817&r=nozend
MySQL Configuration Error:
http://bugs.php.net/fix.php?id=46817&r=mysqlcfg
