For those of us who haven't seen your talk, what's the reason for wanting to add this?
Allen Madsen http://www.allenmadsen.com On Wed, Apr 10, 2019 at 7:50 AM Arjan Scherpenisse <[email protected]> wrote: > Hello all, > > As I discussed with José on ElixirconfEU after my talk on the parser, I > would like to add an option to the tokenizer to be able to still being able > to parse Elixir code without creating atoms. > > The solution would be to have a callback function as an option to the > tokenizer (exposed through `Code.string_to_quoted/2`), that would get > called in the event of the tokenizer encountering an unexisting atom. > Instead of raising, a callback function could be called with the token and > the tokenizer metadata. The callback function returns the data structure > that would be put in the AST instead of the atom. For instance, like in my > talk, an 'atom marker' {:":", "atomname"}. > > The callback gets 4 arguments: > - atom name (string) > - file > - line > - column > > The default behaviour of this function could be to raise an error, like > the existing_atoms_only: true option does now. So two questions before I > implement this: > > 1) How should we call this option? I was thinking of either one of these: > - nonexisting_atom_callback: > - on_nonexisting_atom: > > 2) Should the callback function option be only applicable when > existing_atoms_only: true is passed in as well? Or would it be like this: > > Code.string_to_quoted(string_w_new_atoms) → normal behaviuor, creates atoms > Code.string_to_quoted(string_w_new_atoms, existing_atoms_only: true) → > raises ("builtin" callback behaviour) > Code.string_to_quoted(string_w_new_atoms, nonexisting_atom_callback: > &mycallback/4) → gets called > > in the last case, existing_atoms_only: true is implicit. > However that creates a confusing situation when combining > existing_atoms_only: false + nonexisting_atom_callback . So I prefer to be > explicit: > > Code.string_to_quoted(string_w_new_atoms, nonexisting_atom_callback: > &mycallback/4) → raises RuntimeError > > Code.string_to_quoted(string_w_new_atoms, existing_atoms_only: true, > nonexisting_atom_callback: &mycallback/4) → mycallback gets called > > > Arjan > > > -- > You received this message because you are subscribed to the Google Groups > "elixir-lang-core" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elixir-lang-core/ac2699c7-af26-4b0d-9734-c385aa59aef2%40googlegroups.com > <https://groups.google.com/d/msgid/elixir-lang-core/ac2699c7-af26-4b0d-9734-c385aa59aef2%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAK-y3Cu__02yYX3H5OzXMW_RjXYQGMNwxAqb3cFrCxUWJ5yBpQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
