> What would it emit instead? Just a "number" that happens to contain whitespace? Maybe I am misunderstanding you, though.
The output codes in the state table can trigger emits (i.e. em(i,j,r,c)) or emit vectors (i.e. ev(i,j,r,c)). I'm curious why the state table row for numbers chooses to use the latter. Consider changing all output codes that are 4s (emit vectors) into 2s (emits). Doesn't this split words in exactly the same way? In my original email, I believed the same is true by changing the "0 5" in that row to "6 0". In other words, just let spaces be part of numbers. However, writing this email, I realized that this would let numbers contain trailing whitespace. On Fri, Oct 04, 2019 at 08:14:15AM -0700, Roger Hui wrote: > > emits no vectors > > What would it emit instead? If you emit individual scalar numbers, you > then have to change the parser ( > http://www.jsoftware.com/help/dictionary/dicte.htm) to handle two or more > numbers juxtaposed. > > > > On Thu, Oct 3, 2019 at 11:57 PM ethiejiesa via Programming < > [email protected]> wrote: > > > Just beginning to learn J this week, I find myself binging the > > documentation and have some questions regarding J lexing. In particular, my > > question is about the sample J lexer presented in the dyadic ;: entry of > > the dictionary: > > > > https://www.jsoftware.com/help/dictionary/d332.htm > > > > Essentially, I'm curious why number lexing is choosing to emit vectors. > > Unless I am grossly misunderstanding something, I think we could change the > > state table to one that emits no vectors without affecting word splitting. > > > > Concretely, if we change the line > > > > 1 4 0 5 6 0 6 0 6 0 6 0 6 0 1 0 7 4 NB. 6 num > > > > to > > > > 1 2 6 0 6 0 6 0 6 0 6 0 6 0 1 0 7 2 NB. 6 num > > > > up to word splitting, I believe the state machines are equivalent. > > > > Am I just overlooking something obvious? Or is this simply an aesthetic > > choice? Or, better yet, is it that the trace, more so than the vector of > > words, is what we want in implementing a real tokenizer? > > > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm -- ウィルソン ブランドン (Wilson, Brandon) ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
