> emits no vectors What would it emit instead? If you emit individual scalar numbers, you then have to change the parser ( http://www.jsoftware.com/help/dictionary/dicte.htm) to handle two or more numbers juxtaposed.
On Thu, Oct 3, 2019 at 11:57 PM ethiejiesa via Programming < [email protected]> wrote: > Just beginning to learn J this week, I find myself binging the > documentation and have some questions regarding J lexing. In particular, my > question is about the sample J lexer presented in the dyadic ;: entry of > the dictionary: > > https://www.jsoftware.com/help/dictionary/d332.htm > > Essentially, I'm curious why number lexing is choosing to emit vectors. > Unless I am grossly misunderstanding something, I think we could change the > state table to one that emits no vectors without affecting word splitting. > > Concretely, if we change the line > > 1 4 0 5 6 0 6 0 6 0 6 0 6 0 1 0 7 4 NB. 6 num > > to > > 1 2 6 0 6 0 6 0 6 0 6 0 6 0 1 0 7 2 NB. 6 num > > up to word splitting, I believe the state machines are equivalent. > > Am I just overlooking something obvious? Or is this simply an aesthetic > choice? Or, better yet, is it that the trace, more so than the vector of > words, is what we want in implementing a real tokenizer? > ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
