so, tokens (2+4)+(2+6) is 11 tokens, excluding white space (2 + 4) + ( 2 +
6) is still 11 tokens.........still trying to understand. I should have done
this first.
a token can be each element (word) of a string when talking about characters as
you said below
Armand Postma <[EMAIL PROTECTED]> wrote: Hello,
A token can be a character, but a character is not a token...
Basically a token is a section of a string, for example:
"I am walking home from school"
If we split (tokenize) this line on space-characters we would get the
following 6 tokens:
1. I
2. am
3. walking
4. home
5. from
6. school
As you can see "I" is just 1 character, so a character can be a token, but
one could consider that more a coincidence. Again it depence on your
specific needs,
if you only need characters, you could for example have something like:
A;B;C;D;E;F;G;
If we would tokenize that line using the ; as a splitting character we
would only end up with characters...
The tokenizers (and therefor tokens) are quite commonly used in several
applications, your C++ compiler for example
uses a tokenizer to detect and interpret the commands you put in your
source code, a network application might use it
to tokenizer packages (considering the packages are text rather than
binary). By using a tokenizer it becomes possible that
text commands (tokens) can have variable lengths without the need of
mentioning the length anywhere (besides marking the
end with a splitting character).
A side note about the whole character thing, yes a token can be a
character, but usually a tokenizer (string splitting functions)
spits out strings, so even if the character "I" would be returned by
that routine and the fact that languages like English etc consider "I"
to be a
single character, a programming language will in most cases still
consider "I" a string regardless of it's length. (In C++ that will
be the case if you use the std::string for example).
Hope this helps! :)
Yours Sincerely,
Armand Postma
Robert Ryan schreef:
>
>
> what is the difference between characters and tokens
> can a character be a token, but a token not a character
> thanks
>
> ---------------------------------
> Pinpoint customers who are looking for what you sell.
>
> [Non-text portions of this message have been removed]
>
>
[Non-text portions of this message have been removed]
---------------------------------
Shape Yahoo! in your own image. Join our Network Research Panel today!
[Non-text portions of this message have been removed]
