> The easiest way to implement it in the token reader would probably > be to replace the "reader->state = X" statements with calls to a > private function (e.g. enter_state) that would also store the > token's starting position. Then in flush_token, store this value > when emitting a token. > > Many thanks for the hints :) > > I just merged the type4+tokeniser patch in the trunk. All the unit > tests for the type 4 functions are passing.
The code you committed changes beg_pos in flush_token even if no token is emitted, which could leave it pointing at an unexpected position after an error like EAGAIN. I think that line of code belongs between "*token = new_tok;" and "finish:". Right. Fixed. -- Jose E. Marchesi <[email protected]> http://www.jemarch.net GNU Project http://www.gnu.org
