Well, the good news is that I'm maintaining both start/end tokens in all AST nodes, and inside tokens I maintain line/col/pos/endpos too. And on top of this, there's a comments_before array in all tokens which tell what comments were found before that token (though that's not so useful for perfect reconstitution of whitespace).
Cheers, -M. On Thu, Aug 30, 2012 at 10:03 AM, Marcel Laverdet <[email protected]> wrote: > Yeah I understand the purpose of maintaining comments; I've actually written > a lot of source-to-source transformations myself. The reason I ask is that I > always found it cumbersome to stick non-AST information on the AST itself. > For instance, "for /* this is my loop */ (/* this is my initializer */ var > ii = 0 /* there it is! */;;)". I tortured myself for a long time messing > with maintaining comments and whitespace when I found that the solution is > actually implicit in the token start/end positions. > > If you're rendering the AST to string and you've got the source tokens > attached to each node, you can just look at the original source string to > find the comments and whitespace. Like I said, you should check out the > fibers transformation in Streamline. It does a source-to-source > transformation while maintaining *all* non-semantic source data (whitespace > & comments). > > The other solution I've seen is to maintain a "lastDocBlock" variable in > your tokenizer state. Then when the parser parses a function, you just > attach `lastDocBlock` to the node that way. That's a pretty simple and easy > to implement solution and it handles concerns about maintaining copyright > information, but it won't maintain whitespace or esoteric comments which is > desirable in many situations. > > On Wed, Aug 29, 2012 at 8:07 PM, Scott Taylor <[email protected]> wrote: >> >> Obviously it depends what you are trying to do. Usually compiled >> programming languages strip them out in the tokenizer; for source to source >> translations sometimes you want to keep them in (for instance a header with >> copyright notice when compressing or *all* of them in a case like mine) >> >> Best >> >> Scott >> >> On Aug 29, 2012, at 2:27 PM, Marcel Laverdet <[email protected]> wrote: >> >> Just curious why you need the comments in the AST at all? If you've got >> the start position & length of every token in the AST (much easier to do) >> you implicitly have the comments as well. The "fiber" engine in Streamline >> (https://github.com/Sage/streamlinejs/blob/master/lib/fibers/transform.js) >> does this with really good results. >> >> On Wed, Aug 29, 2012 at 8:37 AM, Mihai Călin Bazon <[email protected]> >> wrote: >>> >>> Well, the code generator doesn't yet have an option to keep comments, >>> but I can add it easily; the harder part was having them in the AST, >>> and that's done. >>> >>> What exactly are you trying to achieve? My understanding is that you >>> compile Lisp to JS (cool!), do you want to be able to do the reverse >>> transform? If so, perhaps a better idea is to generate a source map. >>> (not sure what you need to do though, just guessing) >>> >>> Cheers, >>> -Mihai >>> >>> On Wed, Aug 29, 2012 at 1:52 PM, Scott Taylor <[email protected]> >>> wrote: >>> > Wonderful! I've been working on a project that is sort of like >>> > parenscript >>> > - but much more of a straight javascript in lisp/scheme clothes with a >>> > define-syntax macro system. >>> > >>> > https://github.com/smtlaissezfaire/loop >>> > >>> > I've been hacking on the 1x source of uglify to translate javascript >>> > into a >>> > lispy type system (and back) - but inline comments have been a cause of >>> > concern. Where is the 2.x source at this point? >>> > >>> > Cheers, >>> > >>> > Scott >>> > >>> > On Aug 28, 2012, at 12:56 PM, Mihai Călin Bazon wrote: >>> > >>> > On Tue, Aug 28, 2012 at 5:33 AM, Scott Taylor <[email protected]> >>> > wrote: >>> > >>> > Very cool. What comments in the AST are you going to preserve? >>> > >>> > >>> > The new AST is able to store all comments, and the compressor and code >>> > generator will be able to keep most of them. However, I suspect that >>> > in general people will only need to store copyright notices, and those >>> > usually start with some special marker like "/*!". It'll be easy to >>> > add a configuration option to keep such comments, as long as they're >>> > not in code that's going to be dropped (for example dead code, like, >>> > code that follows a return, throw, break or continue statement). >>> > >>> > Cheers, >>> > -- >>> > Mihai Bazon, >>> > http://mihai.bazon.net/blog >>> > >>> > -- >>> > Job Board: http://jobs.nodejs.org/ >>> > Posting guidelines: >>> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >>> > You received this message because you are subscribed to the Google >>> > Groups "nodejs" group. >>> > To post to this group, send email to [email protected] >>> > To unsubscribe from this group, send email to >>> > [email protected] >>> > For more options, visit this group at >>> > http://groups.google.com/group/nodejs?hl=en?hl=en >>> > >>> > >>> > -- >>> > Job Board: http://jobs.nodejs.org/ >>> > Posting guidelines: >>> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >>> > You received this message because you are subscribed to the Google >>> > Groups "nodejs" group. >>> > To post to this group, send email to [email protected] >>> > To unsubscribe from this group, send email to >>> > [email protected] >>> > For more options, visit this group at >>> > http://groups.google.com/group/nodejs?hl=en?hl=en >>> >>> >>> >>> -- >>> Mihai Bazon, >>> http://mihai.bazon.net/blog >>> >>> -- >>> Job Board: http://jobs.nodejs.org/ >>> Posting guidelines: >>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >>> You received this message because you are subscribed to the Google >>> Groups "nodejs" group. >>> To post to this group, send email to [email protected] >>> To unsubscribe from this group, send email to >>> [email protected] >>> For more options, visit this group at >>> http://groups.google.com/group/nodejs?hl=en?hl=en >> >> >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to >> [email protected] >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en >> >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to >> [email protected] >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en > > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en -- Mihai Bazon, http://mihai.bazon.net/blog -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
