I understand your hesitance.. basically, what I think would be really valuable is a way to transform an AST back to code. That would enable a whole class of refactoring tools: renaming a function (including its call sites), inlining variables, etc, etc. Maybe I'm just spoiled with having used the IntelliJ IDEs for a while.. :-)
I think the formatter_metadata: true annotated AST is already a good step in this direction, the only thing missing there is a way to attach comments. Arjan On Thursday, April 4, 2019 at 9:31:16 AM UTC+2, José Valim wrote: > > For what purpose we would introduce another AST representation? > > I am really hesitant to introduce another interpretation of your AST > unless it is really justified. I am even concerned with exposing the > formatter AST because if there is a formatter bug and we need to change the > formatter AST to fix it, then we will suddenly break people's code and be > unable to improve the formatter, which is the main reason why the formatter > AST exists today in the first place. > > > *José Valim* > www.plataformatec.com.br > Skype: jv.ptec > Founder and Director of R&D > > > On Thu, Apr 4, 2019 at 9:26 AM Arjan Scherpenisse <[email protected] > <javascript:>> wrote: > >> Yes so maybe the route would be to define an "extended AST" format, an >> AST which cannot directly be used to compile or evaluate Elixir >> expressions, but can always be reduced to the "official" AST format? >> >> This extended AST could include everything that the formatter_metadata: >> true option currently includes, but also comment(s) for the nodes (trailing >> comments as well), basically, everything to re-create the original source >> code (given that the original was formatted, we should not add white space >> to this AST). As a bonus we could delay the creation of non-existing atoms >> to the conversion to the "official" AST, so this format could be used for >> DSL that rely on new identifiers but without the atom leakage. >> >> What do you think? >> Arjan >> >> On Wednesday, April 3, 2019 at 10:57:04 PM UTC+2, José Valim wrote: >>> >>> We could and that's the approach we use in the formatter but we can't >>> use it generally because it would break stuff like keyword lists (as the >>> first element is now a tuple __block__ and no longer an atom). >>> >>> >>> *José Valim* >>> www.plataformatec.com.br >>> Skype: jv.ptec >>> Founder and Director of R&D >>> >>> >>> On Wed, Apr 3, 2019 at 10:53 PM Arjan Scherpenisse <[email protected]> >>> wrote: >>> >>>> One thing that comes to mind for "simple value" cases like this: we >>>> could use the existing :__block__ ast node with a single child as nodes >>>> for >>>> simple values which have metadata attributes. AFAIK __block__ ast nodes >>>> now >>>> always have > 1 child. >>>> >>>> This works: >>>> >>>> iex(7)> {:__block__, [], [:a]} |> Macro.to_string >>>> ":a" >>>> >>>> And this as well: >>>> >>>> iex(10)> {:__block__, [], [123]} |> Code.eval_quoted() >>>> {123, []} >>>> >>>> >>>> >>>> On Wednesday, April 3, 2019 at 9:55:22 PM UTC+2, Arjan Scherpenisse >>>> wrote: >>>>> >>>>> Thanks José, >>>>> >>>>> This is indeed a topic of interest to me. Actually, part of my talk in >>>>> Prague coming tuesday will be about this. I'll let it simmer for now and >>>>> maybe we can have a chat during the conference! >>>>> >>>>> cheers, Arjan >>>>> >>>>> On Wednesday, April 3, 2019 at 9:07:24 PM UTC+2, José Valim wrote: >>>>>> >>>>>> Hi Arjan, >>>>>> >>>>>> I don't believe there was any progress here. Storing it in the AST >>>>>> would be hard because some nodes have no metadata slot, so it would be >>>>>> tricky in situations like this: >>>>>> >>>>>> # Let's return just an atom for now >>>>>> :foo >>>>>> >>>>>> >>>>>> However, we do have the ability to return comments from tokenizer and >>>>>> the formatter use this feature: >>>>>> >>>>>> >>>>>> https://github.com/elixir-lang/elixir/blob/master/lib/elixir/lib/code/formatter.ex#L205-L206 >>>>>> >>>>>> Note however those are private APIs but we will be glad to add a new >>>>>> API or an option to one of the functions in the Code module that return >>>>>> the >>>>>> AST + comments from a string. Feel free to experiment with the above and >>>>>> submit a proposal as you see fit. >>>>>> >>>>>> *José Valim* >>>>>> www.plataformatec.com.br >>>>>> Skype: jv.ptec >>>>>> Founder and Director of R&D >>>>>> >>>>>> >>>>>> On Wed, Apr 3, 2019 at 8:27 PM Arjan Scherpenisse < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Steve, just stumbled on this, do you know if anything ever >>>>>>> happened to this? >>>>>>> >>>>>>> On Sunday, September 23, 2018 at 1:10:38 AM UTC+2, Steve Morin wrote: >>>>>>>> >>>>>>>> Tried on Slack but I'll try freenode next >>>>>>>> >>>>>>>> On Fri, Sep 21, 2018 at 10:47 AM OvermindDL1 <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> #elixir-lang on freenode for note* >>>>>>>>> >>>>>>>>> On Friday, September 21, 2018 at 11:47:25 AM UTC-6, OvermindDL1 >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> IRC would be a great place to ask. :-) >>>>>>>>>> >>>>>>>>>> On Friday, September 21, 2018 at 10:36:04 AM UTC-6, Steve Morin >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Anyone know which Elixir AST expert to talk to, about getting >>>>>>>>>>> tips on a starting point? Anyone have suggestions on whom to reach >>>>>>>>>>> out to >>>>>>>>>>> for quick email or call? >>>>>>>>>>> >>>>>>>>>>> On Wed, Sep 19, 2018 at 10:27 AM Steve Morin <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Anyone know which Elixir AST expert to talk to, about getting >>>>>>>>>>>> tips on a starting point? >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Sep 17, 2018 at 12:24 PM Steve Morin < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Yes being part of the metadata would be great. Need to look >>>>>>>>>>>>> around/research and see what other people have done. Just not >>>>>>>>>>>>> losing the >>>>>>>>>>>>> data would be great. >>>>>>>>>>>>> >>>>>>>>>>>>> On Sep 17, 2018, at 12:10, José Valim < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> I don't think we ever stored them in meta. But even if we did, >>>>>>>>>>>>> we would probably have done it in the formatter, and we should >>>>>>>>>>>>> likely move >>>>>>>>>>>>> it somewhere public. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> *José Valim* >>>>>>>>>>>>> www.plataformatec.com.br >>>>>>>>>>>>> Skype: jv.ptec >>>>>>>>>>>>> Founder and Director of R&D >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Sep 17, 2018 at 8:37 PM, Louis Pilfold < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hey >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm a little out of touch with this area, but isn't this what >>>>>>>>>>>>>> we previously did? I remember comments being stored in meta >>>>>>>>>>>>>> during the >>>>>>>>>>>>>> creation of the formatter. Or did I make that up? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>> Louis >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, 17 Sep 2018 at 19:17 José Valim < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I like the metadata idea a lot, thanks. We still need >>>>>>>>>>>>>>> someone to send a detailed proposal, including what will happen >>>>>>>>>>>>>>> with inline >>>>>>>>>>>>>>> comments, comments inside blocks and comments as the last line >>>>>>>>>>>>>>> of a block >>>>>>>>>>>>>>> with no expression afterwards. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We also need a discussion on what will happen with nodes >>>>>>>>>>>>>>> that do not have a metadata entry. Wrapping those in a block is >>>>>>>>>>>>>>> likely >>>>>>>>>>>>>>> enough (but it will break semantics, for example, in keywords >>>>>>>>>>>>>>> lists). So it >>>>>>>>>>>>>>> would need to be an opt-in feature only. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Once all the details are ironed out, then somebody can go >>>>>>>>>>>>>>> ahead and fully implement it. :) >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *José Valim* >>>>>>>>>>>>>>> www.plataformatec.com.br >>>>>>>>>>>>>>> Skype: jv.ptec >>>>>>>>>>>>>>> Founder and Director of R&D >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>>>> Google Groups "elixir-lang-core" group. >>>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails >>>>>>>>>>>>>>> from it, send an email to [email protected] >>>>>>>>>>>>>>> . >>>>>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>>>>> https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4J4%2BcTqcXH_ESJL2_o7rv5OJxkf_yJDpVAH%3DCjH2mmO7w%40mail.gmail.com >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4J4%2BcTqcXH_ESJL2_o7rv5OJxkf_yJDpVAH%3DCjH2mmO7w%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>>>>> . >>>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>>> Google Groups "elixir-lang-core" group. >>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>>>> https://groups.google.com/d/msgid/elixir-lang-core/CABu8xFAk6UQJbbMcD2xkwJFLPM%3DbASSnOqL6ihSdUQEMpuKNGg%40mail.gmail.com >>>>>>>>>>>>>> >>>>>>>>>>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/CABu8xFAk6UQJbbMcD2xkwJFLPM%3DbASSnOqL6ihSdUQEMpuKNGg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>>>> . >>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>> Google Groups "elixir-lang-core" group. >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>>> https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4LmVcoPq7fRkJY5ErJ4N93Bd8-9erEByNETt%2BLN5DZuYA%40mail.gmail.com >>>>>>>>>>>>> >>>>>>>>>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4LmVcoPq7fRkJY5ErJ4N93Bd8-9erEByNETt%2BLN5DZuYA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>>> . >>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Steve Morin | Hacker, Entrepreneur, Startup Advisor >>>>>>>>>>>> twitter.com/SteveMorin | stevemorin.com >>>>>>>>>>>> *Live the dream start a startup. Make the world ... a better >>>>>>>>>>>> place.* >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Steve Morin | Hacker, Entrepreneur, Startup Advisor >>>>>>>>>>> twitter.com/SteveMorin | stevemorin.com >>>>>>>>>>> *Live the dream start a startup. Make the world ... a better >>>>>>>>>>> place.* >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "elixir-lang-core" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/elixir-lang-core/5f1a035a-527d-4b60-9d24-12f376835e01%40googlegroups.com >>>>>>>>> >>>>>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/5f1a035a-527d-4b60-9d24-12f376835e01%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Steve Morin | Hacker, Entrepreneur, Startup Advisor >>>>>>>> twitter.com/SteveMorin | stevemorin.com >>>>>>>> *Live the dream start a startup. Make the world ... a better place.* >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "elixir-lang-core" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/elixir-lang-core/a13ad0ff-4f42-44f9-91f5-8eacaa27450d%40googlegroups.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/a13ad0ff-4f42-44f9-91f5-8eacaa27450d%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elixir-lang-core" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elixir-lang-core/0ab97ebf-39bf-4bcc-a137-49be7ed03f7c%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/elixir-lang-core/0ab97ebf-39bf-4bcc-a137-49be7ed03f7c%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "elixir-lang-core" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elixir-lang-core/db687fa7-7377-4354-b794-7d5be9f6b240%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elixir-lang-core/db687fa7-7377-4354-b794-7d5be9f6b240%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/d6600d04-d3be-438d-9d60-bca4dda0240c%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
