Trying to make a lexer, stops if it hits an operator.

[georgelemon]

Check [std/lexbase](https://nim-lang.org/docs/lexbase.html).

Or even better, use my [toktok](https://github.com/openpeep/toktok) package, a 
generic tokenizer based on Nim `macros`, `lexbase` and `strutils`.

Basic example:
    
    
    import toktok
    
    static:
        Program.settings(uppercase = true, prefix = "TK_")  # limitation: must 
start with TK_
    
    tokens:
      Plus      > '+'
      Minus     > '-'
      Divide    > '/':
        Block_Comment ? '*' .. "*/"
        Inline_Comment ? '/' .. EOL
      Assign    > '='
      Alt_Comment   > '#' .. EOL
      Let       > "let"
      Const     > "const"
      True      > {"TRUE", "True", "true"}
      False     > {"FALSE", "False", "false"}
    
    
    
    Run
    
    
    # sample.txt
    let user = "john"
    
    
    Run

Use `getToken()` to parse each token. This proc returns a `TokenTuple`:
    
    
    (kind: TK_LET, value: "let", wsno: 0, line: 1, col: 0, pos: 0)
    (kind: TK_IDENTIFIER, value: "user", wsno: 1, line: 1, col: 4, pos: 4)
    (kind: TK_ASSIGN, value: "", wsno: 0, line: 1, col: 9, pos: 9)
    (kind: TK_STRING, value: "john", wsno: 1, line: 1, col: 11, pos: 11)
    
    
    
    Run

Full example of toktok in action, implementing a basic `parser` and `AST` 
handler <https://github.com/openpeep/toktok/blob/main/examples/program.nim>