Just beginning to learn J this week, I find myself binging the documentation and have some questions regarding J lexing. In particular, my question is about the sample J lexer presented in the dyadic ;: entry of the dictionary:
https://www.jsoftware.com/help/dictionary/d332.htm Essentially, I'm curious why number lexing is choosing to emit vectors. Unless I am grossly misunderstanding something, I think we could change the state table to one that emits no vectors without affecting word splitting. Concretely, if we change the line 1 4 0 5 6 0 6 0 6 0 6 0 6 0 1 0 7 4 NB. 6 num to 1 2 6 0 6 0 6 0 6 0 6 0 6 0 1 0 7 2 NB. 6 num up to word splitting, I believe the state machines are equivalent. Am I just overlooking something obvious? Or is this simply an aesthetic choice? Or, better yet, is it that the trace, more so than the vector of words, is what we want in implementing a real tokenizer? ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
