> Yesterday I was stung by some odd behaviour in CommonTokenStream, whereby
> I
> was trying to iterate over the token stream looking for tokens of a
> specific
> type. Since ITokenStream doesn't implement IEnumerable, it appears that
> the
> way to do this is as follows:
>
> for (int i = 0; i < tokenStream.Size(); i++)
> {
> IToken token = tokenStream.Get(i);
>
> //... do stuff with token...
> }
>
> However, I was finding that tokenStream.Size() returned 0, despite my
> token
> stream being non-empty. It seems that the underlying stream is lazily
> populated internally, and CommonTokenStream.Size() doesn't trigger a load.
> I
> had to invoke tokenStream.LT(0) to trigger the lazy load prior to looping
> through the tokens. Is this intended behaviour? Does is happen in all of
> the
> different runtimes? If so, it's very counterintuitive.
I believe that this is the general behavior. While a load seems to be sensible
here, please be aware that in ANTLR 3.2 the current CommonTokenStream will
renamed into BufferedTokenStream, as well providing a new implementation for
CommonTokenStream without any buffering. This would result in giving back only
the number of already scanned tokens (if the class does count this - didn't
check it yet).
Quoting your other email: "FWIW, as a relative newcomer to ANTLR, it seems to
me that either CommonTokenStream isn't correctly honouring the ITokenSteam
interface, or else perhaps ITokenStream ought to be updated to formalise
CommonTokenStream's behaviour (though I'm not familiar enough with the other
ITokenStream sub-classes to be sure about this)."
Actually, Size() (which you should change to Count anyway for the newest
runtimes) is defined in IIntStream. Looking at the comment I think a change in
all runtimes is required.
>
> Incidentally, could ITokenStream be updated to implement
> IEnumerable<IToken>
> please? Would others find this useful?
I don't know if there are classes (like the one for ASTs) which wouldn't work
with IToken. Because my development machine is being repaired my capabilities
are stark limited in checking things out. It may be more more sensible to turn
ITokenStream itself generic. But changing interfaces breaks code depending on
it directly. Not all people might use the provided implementing classes. In any
case using generics requires the use of preprocessor symbols as the runtime has
to remain .NET 1.1 compatible until 3.2.
> I don't mind doing the work, but to
> whom should I submit a patch?
Being the C# target maintainer I'm the right man for discussing things
pertaining it, as well for patches, but currently my internet presence will be
spotty due to not having an own computer with internet for now.
Johannes
>
> Chris
--
Psssst! Schon vom neuen GMX MultiMessenger gehört? Der kann`s mit allen:
http://www.gmx.net/de/go/multimessenger01
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"il-antlr-interest" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en
-~----------~----~----~----~------~----~------~--~---
