For what purpose we would introduce another AST representation? I am really hesitant to introduce another interpretation of your AST unless it is really justified. I am even concerned with exposing the formatter AST because if there is a formatter bug and we need to change the formatter AST to fix it, then we will suddenly break people's code and be unable to improve the formatter, which is the main reason why the formatter AST exists today in the first place.
*José Valim* www.plataformatec.com.br Skype: jv.ptec Founder and Director of R&D On Thu, Apr 4, 2019 at 9:26 AM Arjan Scherpenisse <[email protected]> wrote: > Yes so maybe the route would be to define an "extended AST" format, an AST > which cannot directly be used to compile or evaluate Elixir expressions, > but can always be reduced to the "official" AST format? > > This extended AST could include everything that the formatter_metadata: > true option currently includes, but also comment(s) for the nodes (trailing > comments as well), basically, everything to re-create the original source > code (given that the original was formatted, we should not add white space > to this AST). As a bonus we could delay the creation of non-existing atoms > to the conversion to the "official" AST, so this format could be used for > DSL that rely on new identifiers but without the atom leakage. > > What do you think? > Arjan > > On Wednesday, April 3, 2019 at 10:57:04 PM UTC+2, José Valim wrote: >> >> We could and that's the approach we use in the formatter but we can't use >> it generally because it would break stuff like keyword lists (as the first >> element is now a tuple __block__ and no longer an atom). >> >> >> *José Valim* >> www.plataformatec.com.br >> Skype: jv.ptec >> Founder and Director of R&D >> >> >> On Wed, Apr 3, 2019 at 10:53 PM Arjan Scherpenisse <[email protected]> >> wrote: >> >>> One thing that comes to mind for "simple value" cases like this: we >>> could use the existing :__block__ ast node with a single child as nodes for >>> simple values which have metadata attributes. AFAIK __block__ ast nodes now >>> always have > 1 child. >>> >>> This works: >>> >>> iex(7)> {:__block__, [], [:a]} |> Macro.to_string >>> ":a" >>> >>> And this as well: >>> >>> iex(10)> {:__block__, [], [123]} |> Code.eval_quoted() >>> {123, []} >>> >>> >>> >>> On Wednesday, April 3, 2019 at 9:55:22 PM UTC+2, Arjan Scherpenisse >>> wrote: >>>> >>>> Thanks José, >>>> >>>> This is indeed a topic of interest to me. Actually, part of my talk in >>>> Prague coming tuesday will be about this. I'll let it simmer for now and >>>> maybe we can have a chat during the conference! >>>> >>>> cheers, Arjan >>>> >>>> On Wednesday, April 3, 2019 at 9:07:24 PM UTC+2, José Valim wrote: >>>>> >>>>> Hi Arjan, >>>>> >>>>> I don't believe there was any progress here. Storing it in the AST >>>>> would be hard because some nodes have no metadata slot, so it would be >>>>> tricky in situations like this: >>>>> >>>>> # Let's return just an atom for now >>>>> :foo >>>>> >>>>> >>>>> However, we do have the ability to return comments from tokenizer and >>>>> the formatter use this feature: >>>>> >>>>> >>>>> https://github.com/elixir-lang/elixir/blob/master/lib/elixir/lib/code/formatter.ex#L205-L206 >>>>> >>>>> Note however those are private APIs but we will be glad to add a new >>>>> API or an option to one of the functions in the Code module that return >>>>> the >>>>> AST + comments from a string. Feel free to experiment with the above and >>>>> submit a proposal as you see fit. >>>>> >>>>> *José Valim* >>>>> www.plataformatec.com.br >>>>> Skype: jv.ptec >>>>> Founder and Director of R&D >>>>> >>>>> >>>>> On Wed, Apr 3, 2019 at 8:27 PM Arjan Scherpenisse <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Steve, just stumbled on this, do you know if anything ever >>>>>> happened to this? >>>>>> >>>>>> On Sunday, September 23, 2018 at 1:10:38 AM UTC+2, Steve Morin wrote: >>>>>>> >>>>>>> Tried on Slack but I'll try freenode next >>>>>>> >>>>>>> On Fri, Sep 21, 2018 at 10:47 AM OvermindDL1 <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> #elixir-lang on freenode for note* >>>>>>>> >>>>>>>> On Friday, September 21, 2018 at 11:47:25 AM UTC-6, OvermindDL1 >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> IRC would be a great place to ask. :-) >>>>>>>>> >>>>>>>>> On Friday, September 21, 2018 at 10:36:04 AM UTC-6, Steve Morin >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Anyone know which Elixir AST expert to talk to, about getting >>>>>>>>>> tips on a starting point? Anyone have suggestions on whom to reach >>>>>>>>>> out to >>>>>>>>>> for quick email or call? >>>>>>>>>> >>>>>>>>>> On Wed, Sep 19, 2018 at 10:27 AM Steve Morin <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Anyone know which Elixir AST expert to talk to, about getting >>>>>>>>>>> tips on a starting point? >>>>>>>>>>> >>>>>>>>>>> On Mon, Sep 17, 2018 at 12:24 PM Steve Morin <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Yes being part of the metadata would be great. Need to look >>>>>>>>>>>> around/research and see what other people have done. Just not >>>>>>>>>>>> losing the >>>>>>>>>>>> data would be great. >>>>>>>>>>>> >>>>>>>>>>>> On Sep 17, 2018, at 12:10, José Valim < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>> I don't think we ever stored them in meta. But even if we did, >>>>>>>>>>>> we would probably have done it in the formatter, and we should >>>>>>>>>>>> likely move >>>>>>>>>>>> it somewhere public. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> *José Valim* >>>>>>>>>>>> www.plataformatec.com.br >>>>>>>>>>>> Skype: jv.ptec >>>>>>>>>>>> Founder and Director of R&D >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Sep 17, 2018 at 8:37 PM, Louis Pilfold < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hey >>>>>>>>>>>>> >>>>>>>>>>>>> I'm a little out of touch with this area, but isn't this what >>>>>>>>>>>>> we previously did? I remember comments being stored in meta >>>>>>>>>>>>> during the >>>>>>>>>>>>> creation of the formatter. Or did I make that up? >>>>>>>>>>>>> >>>>>>>>>>>>> Cheers, >>>>>>>>>>>>> Louis >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, 17 Sep 2018 at 19:17 José Valim < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I like the metadata idea a lot, thanks. We still need someone >>>>>>>>>>>>>> to send a detailed proposal, including what will happen with >>>>>>>>>>>>>> inline >>>>>>>>>>>>>> comments, comments inside blocks and comments as the last line >>>>>>>>>>>>>> of a block >>>>>>>>>>>>>> with no expression afterwards. >>>>>>>>>>>>>> >>>>>>>>>>>>>> We also need a discussion on what will happen with nodes that >>>>>>>>>>>>>> do not have a metadata entry. Wrapping those in a block is >>>>>>>>>>>>>> likely enough >>>>>>>>>>>>>> (but it will break semantics, for example, in keywords lists). >>>>>>>>>>>>>> So it would >>>>>>>>>>>>>> need to be an opt-in feature only. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Once all the details are ironed out, then somebody can go >>>>>>>>>>>>>> ahead and fully implement it. :) >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> *José Valim* >>>>>>>>>>>>>> www.plataformatec.com.br >>>>>>>>>>>>>> Skype: jv.ptec >>>>>>>>>>>>>> Founder and Director of R&D >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>>> Google Groups "elixir-lang-core" group. >>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>>>> https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4J4%2BcTqcXH_ESJL2_o7rv5OJxkf_yJDpVAH%3DCjH2mmO7w%40mail.gmail.com >>>>>>>>>>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4J4%2BcTqcXH_ESJL2_o7rv5OJxkf_yJDpVAH%3DCjH2mmO7w%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>>>> . >>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>> Google Groups "elixir-lang-core" group. >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>>> https://groups.google.com/d/msgid/elixir-lang-core/CABu8xFAk6UQJbbMcD2xkwJFLPM%3DbASSnOqL6ihSdUQEMpuKNGg%40mail.gmail.com >>>>>>>>>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/CABu8xFAk6UQJbbMcD2xkwJFLPM%3DbASSnOqL6ihSdUQEMpuKNGg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>>> . >>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>> Google Groups "elixir-lang-core" group. >>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>> https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4LmVcoPq7fRkJY5ErJ4N93Bd8-9erEByNETt%2BLN5DZuYA%40mail.gmail.com >>>>>>>>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4LmVcoPq7fRkJY5ErJ4N93Bd8-9erEByNETt%2BLN5DZuYA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>> . >>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Steve Morin | Hacker, Entrepreneur, Startup Advisor >>>>>>>>>>> twitter.com/SteveMorin | stevemorin.com >>>>>>>>>>> *Live the dream start a startup. Make the world ... a better >>>>>>>>>>> place.* >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Steve Morin | Hacker, Entrepreneur, Startup Advisor >>>>>>>>>> twitter.com/SteveMorin | stevemorin.com >>>>>>>>>> *Live the dream start a startup. Make the world ... a better >>>>>>>>>> place.* >>>>>>>>>> >>>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "elixir-lang-core" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/elixir-lang-core/5f1a035a-527d-4b60-9d24-12f376835e01%40googlegroups.com >>>>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/5f1a035a-527d-4b60-9d24-12f376835e01%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Steve Morin | Hacker, Entrepreneur, Startup Advisor >>>>>>> twitter.com/SteveMorin | stevemorin.com >>>>>>> *Live the dream start a startup. Make the world ... a better place.* >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "elixir-lang-core" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/elixir-lang-core/a13ad0ff-4f42-44f9-91f5-8eacaa27450d%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/a13ad0ff-4f42-44f9-91f5-8eacaa27450d%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elixir-lang-core" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elixir-lang-core/0ab97ebf-39bf-4bcc-a137-49be7ed03f7c%40googlegroups.com >>> <https://groups.google.com/d/msgid/elixir-lang-core/0ab97ebf-39bf-4bcc-a137-49be7ed03f7c%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- > You received this message because you are subscribed to the Google Groups > "elixir-lang-core" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elixir-lang-core/db687fa7-7377-4354-b794-7d5be9f6b240%40googlegroups.com > <https://groups.google.com/d/msgid/elixir-lang-core/db687fa7-7377-4354-b794-7d5be9f6b240%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4K-CezhUcO%3D6cJ_v2x7kkCjcMwu-UmS1TLFqnisnPXz8Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
