> What would it emit instead? Words that just happen to contain spaces. I just realized, however, that my original email example lets number words contain trailing spaces. So, I agree that the cell "0 5" in the state table needs to be what it is. However, if we're in state num and encounter a quote or "other" character, I still don't grok why ev (emit vector) is used instead of em (emit). Is there a case where we have a number token containing a quote character, or likewise for "other" characters?
On Fri, Oct 04, 2019 at 08:14:15AM -0700, Roger Hui wrote: > > emits no vectors > > What would it emit instead? If you emit individual scalar numbers, you > then have to change the parser ( > http://www.jsoftware.com/help/dictionary/dicte.htm) to handle two or more > numbers juxtaposed. > > > > On Thu, Oct 3, 2019 at 11:57 PM ethiejiesa via Programming < > [email protected]> wrote: > > > Just beginning to learn J this week, I find myself binging the > > documentation and have some questions regarding J lexing. In particular, my > > question is about the sample J lexer presented in the dyadic ;: entry of > > the dictionary: > > > > https://www.jsoftware.com/help/dictionary/d332.htm > > > > Essentially, I'm curious why number lexing is choosing to emit vectors. > > Unless I am grossly misunderstanding something, I think we could change the > > state table to one that emits no vectors without affecting word splitting. > > > > Concretely, if we change the line > > > > 1 4 0 5 6 0 6 0 6 0 6 0 6 0 1 0 7 4 NB. 6 num > > > > to > > > > 1 2 6 0 6 0 6 0 6 0 6 0 6 0 1 0 7 2 NB. 6 num > > > > up to word splitting, I believe the state machines are equivalent. > > > > Am I just overlooking something obvious? Or is this simply an aesthetic > > choice? Or, better yet, is it that the trace, more so than the vector of > > words, is what we want in implementing a real tokenizer? > > > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm -- ウィルソン ブランドン (Wilson, Brandon) ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
