Hello, A token can be a character, but a character is not a token...
Basically a token is a section of a string, for example: "I am walking home from school" If we split (tokenize) this line on space-characters we would get the following 6 tokens: 1. I 2. am 3. walking 4. home 5. from 6. school As you can see "I" is just 1 character, so a character can be a token, but one could consider that more a coincidence. Again it depence on your specific needs, if you only need characters, you could for example have something like: A;B;C;D;E;F;G; If we would tokenize that line using the ; as a splitting character we would only end up with characters... The tokenizers (and therefor tokens) are quite commonly used in several applications, your C++ compiler for example uses a tokenizer to detect and interpret the commands you put in your source code, a network application might use it to tokenizer packages (considering the packages are text rather than binary). By using a tokenizer it becomes possible that text commands (tokens) can have variable lengths without the need of mentioning the length anywhere (besides marking the end with a splitting character). A side note about the whole character thing, yes a token can be a character, but usually a tokenizer (string splitting functions) spits out strings, so even if the character "I" would be returned by that routine and the fact that languages like English etc consider "I" to be a single character, a programming language will in most cases still consider "I" a string regardless of it's length. (In C++ that will be the case if you use the std::string for example). Hope this helps! :) Yours Sincerely, Armand Postma Robert Ryan schreef: > > > what is the difference between characters and tokens > can a character be a token, but a token not a character > thanks > > --------------------------------- > Pinpoint customers who are looking for what you sell. > > [Non-text portions of this message have been removed] > > [Non-text portions of this message have been removed]
