Re: syntax-local-binding
Hello Mark, On Wed 25 Jan 2012 03:30, Mark H Weaver m...@netris.org writes: Perhaps, though, at this point we're just going to have to agree to disagree; This is a euphemism for sorry, but we're doing it my way. Just a few days ago, you said: On Sun 22 Jan 2012 13:23, Mark H Weaver m...@netris.org writes: If Andy finishes his version of `local-eval', that's all well and good, and I'm happy to use his version. You got a fairly explicit NAK from me on the-environment in psyntax: beginning with suggestions for a different approach, continuing where I implemented those suggestions, and finally just a few days ago. (I.e., I do not consent to the-environment in psyntax, given the existence of other possibilities.) But, you are interested in local-eval in 2.0.4, and you consented to this approach. Please. Let us focus on solutions. Regards, Andy -- http://wingolog.org/
Re: syntax-local-binding
Hello Happy Guilers! :-) Sorry for remaining silent in this heated thread (I spent my spare time on other practical issues for 2.0.4, and felt I lacked the competence.) My overall feeling is that providing a 1.8-compatible ‘local-eval’ in 2.0 is great, but that it’s essentially one bug to be fixed among many others. Thus, I appreciate all the work Andy and you have put into it, and I’m glad it will be helpful to Guile users such as Lilypond. However, I think, that we should keep in mind that Guile is not just about ‘local-eval’, and that there are other great things to work on together. As for the technical aspects: Mark H Weaver m...@netris.org skribis: Andy Wingo wi...@pobox.com writes: On Tue 24 Jan 2012 14:25, Mark H Weaver m...@netris.org writes: Andy Wingo wi...@pobox.com writes: None of the interfaces that I proposed leak internal psyntax representations. `syntax-local-binding' leaks the internal representations used for bindings. You mean, whether something is a lexical, or a macro, or a global, or whatever; OK. That's actually not what I meant, although I'm not convinced that we fully understand the implications of exposing even that much. One thing that is already clear is that `identifier-syntax' and `local-eval' were previously capable of emulating variables perfectly before, whereas in the presence of `syntax-local-binding' they no longer are. This is a perfect example of how added flexibility in one aspect can lead to _reduced_ flexibility in other aspects. I think we need more time to consider the implications of this. I agree. Yet, it’s the kind of API that is useful internally for different purposes beyond ‘local-eval’, as this discussion showed, right? So, IIUC, the tension is: 1. Such as API is desirable for Guile-internal consumption. 2. Exposing it to users may restrict our ability to change the implementation in the future, just like ‘the-environment’ did. I haven’t fully thought about it, but one possible trade-off would be to mark ‘syntax-local-binding’ co. as internal, somehow arrange to make them visible only from some (system ...) module, and document them in the “Implementation” chapter. WDYT? Thanks, Ludo’.
Re: syntax-local-binding
I have nothing new to say here, so I'll spare you all my increasingly frustrated repetitions. I need to walk away for a while. Do what you feel is best. Good luck. Mark
Re: syntax-local-binding
Peter TB Brett pe...@peter-b.co.uk writes: It seems pretty clear to me that the only (debatable) downside to using Mark's implementation is that some definitions end up in the wrong module, while your implementation has several potentially *major* problems (including the necessity of providing universally unique gensyms) which Mark has managed to avoid. Let me just repeat that I consider universally unique gensyms a recipe for trouble. Having a compiler _depend_ on creating unreproducible output is going to be in the not really fun category for a _lot_ of things. I don't have much of a clue about the other differences. But this one _really_ comes at a dear price in its implications. -- David Kastrup
Re: syntax-local-binding
Hi Mark, On Tue 24 Jan 2012 03:11, Mark H Weaver m...@netris.org writes: you seem unabashedly content to lock us into using psyntax forever, This statement is an exaggeration. While I am content with psyntax now, all change is possible, with time. While it's important to think of the future (and the past), one must not forget about the present :) It concerns me when I see internal psyntax representations exported in our API. None of the interfaces that I proposed leak internal psyntax representations. syntax-local-binding provides binding information for an identifier. Racket provides similar procedures, and does not use psyntax. Therefore this information does not tie us to the psyntax implementation. syntax-locally-bound-identifiers can be implemented in any expander. It provides syntax objects. Syntax objects are not a psyntax implementation detail. syntax-module is a simple accessor. Racket provides the same accessor. Therefore it does not leak psyntax implementation details. If we can already foresee the need to deprecate an interface, wouldn't it be better not to add it in the first place? I don't see the need to deprecate them now, not more than any other identifier that we export. I just have one final request: please at least change the lexical environments in your `local-eval' implementation to use the future-proof `evaluator procedure' representation, as I have done in mine. For the reasons I mentioned in my mail yesterday at 12:52 UTC, I really don't see the point, as we have more effective means of dealing with future change than introducing an abstraction there. But if it will make you happy, sure. I'm quite tired of this topic ;-) Regards, Andy -- http://wingolog.org/
Re: syntax-local-binding
Andy Wingo wi...@pobox.com writes: On Tue 24 Jan 2012 11:30, Peter TB Brett pe...@peter-b.co.uk writes: It seems pretty clear to me that the only (debatable) downside to using Mark's implementation is that some definitions end up in the wrong module, while your implementation has several potentially *major* problems (including the necessity of providing universally unique gensyms) Let's be clear here: the universally-unique gensym issue is something that Guile *already* has, in version 2.0.0, 2.0.1, etc. I don't see why we need universally-unique gensyms unless your approach to `local-eval' is used. I've already explained why they are not needed for macros compiled in another session. It concerns me when I see internal psyntax representations exported in our API. None of the interfaces that I proposed leak internal psyntax representations. `syntax-local-binding' leaks the internal representations used for bindings. I gave an example where this would constrain our ability to change the binding representation for syntactic keywords, and your response was to point out that my particular example could be done in a different way without changing the representation. Your response demonstrated the weakness of my particular example, while underscoring my main point: that this constrains our internal representation choices. I just have one final request: please at least change the lexical environments in your `local-eval' implementation to use the future-proof `evaluator procedure' representation, as I have done in mine. For the reasons I mentioned in my mail yesterday at 12:52 UTC, I really don't see the point, as we have more effective means of dealing with future change than introducing an abstraction there. Your suggested more effective means is to introduce a new type lexical-environment-2 and continue supporting lexical-environment. No thanks. But if it will make you happy, sure. I'm quite tired of this topic ;-) Yes, it would make me happy, thank you. And believe me, I'm tired of this topic too. I thought I was mostly finished working on `local-eval' three weeks ago, when I produced my simple patch, which I _still_ think is superior to yours in the most important respect, namely in the commitments that it forces us to make. Regards, Mark
Re: syntax-local-binding
Hello, If we can already foresee the need to deprecate an interface, wouldn't it be better not to add it in the first place? I don't see the need to deprecate them now, not more than any other identifier that we export. I think this may be the key to this argument. There are two separate questions being debated as one question. Here they are: 1. Do we forsee a need to deprecate the syntax-local-binding functionality in the future, for instance to move away from psyntax? 2. Is this version of syntax-local-binding the best interface to this functionality? They are related questions, but they are distinct because someone could believe that it is good to expose this functionality but that the current syntax-local-bindings is not the best interface for us. I could be wrong, but I think that Andy answers maybe, but not for a long time to 1, and therefore thinks it's fine to include syntax-local-binding. Mark answers maybe, but definitely needs more thought before we make a commitment to 2, and therefore does not want to include syntax-local-binding. These are not contradictory positions. (Some of their other positions are contradictory, though :-) ). However, it does make me think that we should discuss the interface to syntax-local-binding more before releasing it (but don't take this too seriously, because I didn't follow the earlier threads much). Noah
mark uniqueness (Was: Re: syntax-local-binding)
On Tue 24 Jan 2012 14:25, Mark H Weaver m...@netris.org writes: I don't see why we need universally-unique gensyms I've already explained why they are not needed for macros compiled in another session. Ah, I forgot to reply to that. I found it: On Mon 16 Jan 2012 14:28, Mark H Weaver m...@netris.org writes: The reason it has not been a problem with macros is that, within a top-level macro (which are the only ones used across Guile sessions), the only syntax-objects that can be meaningfully _introduced_ into the expansion are top-level/module bindings. But these bindings have no associated labels or gensyms, because they're not in the wrap. See how this is a problem now where it wasn't before? Or am I missing something? Either you are missing something, or I am, or both of us -- that much is clear ;-) Psyntax associates marks with every identifier. Two identifiers are equal if they are symbolically equal, and they have the same marks. It would break hygiene if two identifiers that didn't come from the same place accidentally had the same marks. A fresh mark is placed on syntax returned from a macro expander, if the syntax was not present in the input. An easy way to do this would be simply: (define-syntax-rule (fresh-identifier) #'x) (define my-id (fresh-identifier)) All you need to do is to introduce that binding into a macro, and you might alias some other binding, because you have serialized the symbol and marks into a compiled file. This is admittedly far-fetched. But it can happen, and at the top-level. For example, our old friend: (define-syntax-rule (define-const x val) (begin (define t val) (define-syntax x (identifier-syntax t Here, `t' will have a fresh mark. Now, if in one compilation unit, I do: (define-const x 10) And in another, I do: (let ((t 20)) x) = ? You would expect the result to be 20. But I think it could be 20, if the marks on the two ts happened to collide. Am I missing something? :-) Andy -- http://wingolog.org/
Re: syntax-local-binding
Hello Mark :) Thanks again for your deep thoughts. This conversation is a bit stressful for the both of us, but we wouldn't be having it if we didn't both care about Guile. In the spirit of diffusing tension here, feel free to imagine all of my words as coming from Mr. Collins for the duration of this thread ;-) On Tue 24 Jan 2012 14:25, Mark H Weaver m...@netris.org writes: Andy Wingo wi...@pobox.com writes: None of the interfaces that I proposed leak internal psyntax representations. `syntax-local-binding' leaks the internal representations used for bindings. You mean, whether something is a lexical, or a macro, or a global, or whatever; OK. I think that leak is the wrong word here: leaks are inadvertent, whereas providing this information is what this function was designed to do; and furthermore it's not a detail of psyntax (cf Racket which does not use psyntax). Let me offer another example of its utility: writing a macro stepper. Again, for an example I'll have to link to the PLT folks' great work: http://docs.racket-lang.org/macro-debugger/index.html With syntax-local-binding, syntax-locally-bound-identifiers, and a couple hooks that get fired when a macro is expanded and reconstructed, you could implement a macro stepper for Guile. And, once we provide those hooks, you can implement this in a module. Pretty sweet, if you ask me! Perhaps, though, at this point we're just going to have to agree to disagree; surely we have plumbed the depths sufficiently. It's not satisfying, but I think we both made a great effort to communicate and convince -- and in some cases, to work on each other's code. We really need to move on here. I will re-post my patches taking into account your comments regarding the form of the environments. Regards, Andy -- http://wingolog.org/
Re: syntax-local-binding
Andy Wingo wi...@pobox.com writes: In the spirit of diffusing tension here, feel free to imagine all of my words as coming from Mr. Collins for the duration of this thread ;-) Hehe :) On Tue 24 Jan 2012 14:25, Mark H Weaver m...@netris.org writes: Andy Wingo wi...@pobox.com writes: None of the interfaces that I proposed leak internal psyntax representations. `syntax-local-binding' leaks the internal representations used for bindings. You mean, whether something is a lexical, or a macro, or a global, or whatever; OK. That's actually not what I meant, although I'm not convinced that we fully understand the implications of exposing even that much. One thing that is already clear is that `identifier-syntax' and `local-eval' were previously capable of emulating variables perfectly before, whereas in the presence of `syntax-local-binding' they no longer are. This is a perfect example of how added flexibility in one aspect can lead to _reduced_ flexibility in other aspects. I think we need more time to consider the implications of this. However, the more serious problem, and the one I was actually talking about, is the _second_ value returned by `syntax-local-binding', which exposes the representations of bindings stored in the cdrs of the psyntax `r' alist. Let me offer another example of its utility: writing a macro stepper. It's not the least bit surprising that exposing internal implementation details enables the creation of all kinds of nifty things as external programs that would ordinarily need to be internal. This is perfectly obvious. However, as you wisely said to me when I posted my first `local-eval' evaluator-only implementation, this stuff has a cost, and it has to justify itself. I assumed you meant the maintenance cost of supporting this advanced functionality indefinitely, and the constraints that it places on the freedom of future implementors. Was I right? In retrospect, based on your recent behavior, I wonder if that's what you meant or if you were talking about something completely different, because you seem to have completely abandoned your original restraint, and have now gone much farther than I would ever dare to go. Indeed, as you can see, I am very unhappy about how far you have gone with this. Perhaps, though, at this point we're just going to have to agree to disagree; This is a euphemism for sorry, but we're doing it my way. Multiple people here have expressed grave concerns about your approach, and not a single person has publicly expressed their support for adding all of these new interfaces you've designed. Even Ludovic has remained silent. And yet you apparently intend to rush all of this stuff into 2.0.4. After all, nothing could be worse than having `the-environment' in psyntax, even for single release. Is this your idea of a consensus process, which you so often advocate? Mark
Re: syntax-local-binding
Heya Mark, On Fri 20 Jan 2012 23:03, Mark H Weaver m...@netris.org writes: (let ((x 1)) (syntax-local-binding #'x)) is not equivalent to: (let ((x 1)) (local-eval '(syntax-local-binding #'x) (the-environment))) Indeed; bummer! I think, though, that this is simply a consequence of giving more power to macro writers. It is analogous in some ways to the changes that identifier-syntax introduce into macro writing: with identifier-syntax, one can no longer write a code walker with syntax-rules pattern matching, as single identifiers may expand out to complicated expressions, possibly even with side effects. Why do you think that? The procedures do carry metadata; I understood that that was your strategy, to use the serialization of the syntax-rules form in the procedure metadata. Well, this was in the context of a new strategy where psyntax would include a new core form called `call-with-current-local-expander' that calls its parameter (a procedure or macro) with a procedure that accepts an expression and returns an expanded form. In this case, the most straightforward implementation would simply serialize the (r w mod) structures directly. Toward that end, I was thinking it would be nice to keep those structures serializable. The only part that's not currently serializable are the transformer procedures for local macros. Thus the change in representation. I have been staring at this empty page here for a little while, writing and re-writing, but I can't get over a feeling that I really don't want this kind of work in psyntax itself. Who knows, maybe you have really convincing arguments here, but this particular argument should not be driving a decision about e.g. including syntax-local-binding or not. That sounds negative, and in a way of course it is -- but still, I'd much rather enable people to make powerful syntactic abstractions like local-eval outside psyntax. Syntax-parse, for example, if it ever lands, will land in the form of a module In this case there are lots of strategies you could use. We could change psyntax to embed the syntax objects in the procedure meta-data, like I said. Ice-9 local-eval could #:replace its own syntax-rules. We could (and probably should) do procedure serialization. Regards, Andy -- http://wingolog.org/
Re: syntax-local-binding
Andy Wingo wi...@pobox.com writes: On Fri 20 Jan 2012 23:03, Mark H Weaver m...@netris.org writes: (let ((x 1)) (syntax-local-binding #'x)) is not equivalent to: (let ((x 1)) (local-eval '(syntax-local-binding #'x) (the-environment))) Indeed; bummer! I think, though, that this is simply a consequence of giving more power to macro writers. It is analogous in some ways to the changes that identifier-syntax introduce into macro writing: with identifier-syntax, one can no longer write a code walker with syntax-rules pattern matching, as single identifiers may expand out to complicated expressions, possibly even with side effects. This is false. Macros are always expanded _before_ any of their arguments are expanded. Therefore, a code walker sees the unexpanded forms, including any simulated variables bound by identifier-syntax. Why do you think that? The procedures do carry metadata; I understood that that was your strategy, to use the serialization of the syntax-rules form in the procedure metadata. Well, this was in the context of a new strategy where psyntax would include a new core form called `call-with-current-local-expander' that calls its parameter (a procedure or macro) with a procedure that accepts an expression and returns an expanded form. In this case, the most straightforward implementation would simply serialize the (r w mod) structures directly. Toward that end, I was thinking it would be nice to keep those structures serializable. The only part that's not currently serializable are the transformer procedures for local macros. Thus the change in representation. I have been staring at this empty page here for a little while, writing and re-writing, but I can't get over a feeling that I really don't want this kind of work in psyntax itself. Your priorities are reversed from what they ought to be. What you _should_ be worried about is making commitments in our API that we must continue to support forever. This is a _real_ problem, since it constrains our ability to modify our implementation in the future. Putting the `the-environment' in psyntax is, at worst, a stylistic issue. Whether it belongs there is a matter of taste, but however strongly you may feel about that, it is a purely _internal_ implementation issue. The really important thing is that it commits us to _nothing_. There's nothing stopping us from radically reimplementing it later. In particular, there's nothing stopping us from moving it out of psyntax later. Guile has been in existence for a couple of decades already, and I hope that it will be actively used for many decades to come. With that in mind, please consider the long view. One of the reasons Scheme has lasted so long is because it tries exceptionally hard to hide internal implementation details. Implementations that expose too much internal detail may derive a short-term benefit from doing so, but it comes at the price of eventual calcification: there comes a time when implementations that expose too much become unable to make significant internal structural changes. Please consider this. I feel that your mind has become closed to my arguments. Mark
Re: syntax-local-binding
Hi Mark, On Mon 23 Jan 2012 22:03, Mark H Weaver m...@netris.org writes: Andy Wingo wi...@pobox.com writes: with identifier-syntax, one can no longer write a code walker with syntax-rules pattern matching, as single identifiers may expand out to complicated expressions, possibly even with side effects. This is false. Macros are always expanded _before_ any of their arguments are expanded. Therefore, a code walker sees the unexpanded forms, including any simulated variables bound by identifier-syntax. I'm not sure we're talking about the same thing here; do see Alex Shinn's NAK on R6RS: http://www.r6rs.org/ratification/results.html#X70 While I disagree with his assessment of the identifier-syntax tradeoff, I think he correctly identifies it as a tradeoff. Why do you think that? The procedures do carry metadata; I understood that that was your strategy, to use the serialization of the syntax-rules form in the procedure metadata. Well, this was in the context of a new strategy where psyntax would include a new core form called `call-with-current-local-expander' that calls its parameter (a procedure or macro) with a procedure that accepts an expression and returns an expanded form. In this case, the most straightforward implementation would simply serialize the (r w mod) structures directly. Toward that end, I was thinking it would be nice to keep those structures serializable. The only part that's not currently serializable are the transformer procedures for local macros. Thus the change in representation. I have been staring at this empty page here for a little while, writing and re-writing, but I can't get over a feeling that I really don't want this kind of work in psyntax itself. Your priorities are reversed from what they ought to be. What you _should_ be worried about is making commitments in our API that we must continue to support forever. This is a _real_ problem, since it constrains our ability to modify our implementation in the future. I know I'm going to sound like Mr. Collins in Pride and Prejudice here, but I flatter myself that I know a thing or two about managing change -- I mean, replacing the lazy-memoizing evaluator with the compiler, retrofitting psyntax into Guile, the whole subr mess, etc. There is always room to improve, of course, as in all human endeavor, but for now it does seem that Guile is getting more powerful _and_ more limpid over time. But frankly though, regarding change, while we do need the freedom to modify some things, some other practical freedoms just don't make the cost/benefit cut, for me. For example, considering replacing psyntax, which seems to be in the back of your mind here. This conservatism is preventing Guile from adding features. And we do need features -- local-eval and syntax-parse among them. Putting the `the-environment' in psyntax is, at worst, a stylistic issue. Whether it belongs there is a matter of taste, but however strongly you may feel about that, it is a purely _internal_ implementation issue. The really important thing is that it commits us to _nothing_. There's nothing stopping us from radically reimplementing it later. In particular, there's nothing stopping us from moving it out of psyntax later. Apart from the fact with `the-environment' in psyntax, it's in the default environment, of course; though with autoloads one can get around that... With `the-environment' in a module, in 2.0.4 we could have three functions added to psyntax: syntax-local-binding, syntax-locally-bound-identifiers, and syntax-module. The first and the third have precedent in Racket (whose hackers have been able to do significantly awesome stuff). The second is strange, but seems OK. I'm OK with them. But what if they're the wrong interface? Well, then we use the normal deprecation mechanism to get rid of them, eventually (in the 2.2 series, for example). We learned something. Users get warned off the code. Life goes on. Guile has been in existence for a couple of decades already, and I hope that it will be actively used for many decades to come. Hear, hear. With that in mind, please consider the long view. One of the reasons Scheme has lasted so long is because it tries exceptionally hard to hide internal implementation details. Implementations that expose too much internal detail may derive a short-term benefit from doing so, but it comes at the price of eventual calcification: there comes a time when implementations that expose too much become unable to make significant internal structural changes. Please consider this. I feel that your mind has become closed to my arguments. I really do value your work, and your words, Mark. Besides that personal appreciation, I think you're doing good work for Guile. We happen to disagree here on a matter of implementation. OK. It's a feature we have worked on sufficiently that it should probably make it into 2.0.4. OK. One of us
Re: syntax-local-binding
Andy Wingo wi...@pobox.com writes: Your priorities are reversed from what they ought to be. What you _should_ be worried about is making commitments in our API that we must continue to support forever. This is a _real_ problem, since it constrains our ability to modify our implementation in the future. I know I'm going to sound like Mr. Collins in Pride and Prejudice here, but I flatter myself that I know a thing or two about managing change -- I mean, replacing the lazy-memoizing evaluator with the compiler, retrofitting psyntax into Guile, the whole subr mess, etc. These are certainly impressive accomplishments, and I salute you for this excellent work! :) However, these accomplishments do not demonstrate that you understand the importance of hiding implementation details so that future Guile hackers can make similar transformations a decade or two from now. For example, you seem unabashedly content to lock us into using psyntax forever, despite the fact that it has known deficiencies having to do with its phase story, as well as limitations in its handling of hygiene in complex macros. (c.f. Improved hygiene, SRFI 72). I don't mean to suggest that we should replace psyntax anytime soon, but we might want to replace it in a decade or two. Therefore, it concerns me when I see internal psyntax representations exported in our API. But frankly though, regarding change, while we do need the freedom to modify some things, some other practical freedoms just don't make the cost/benefit cut, for me. I agree that sometimes practical necessity outweighs the need to hide implementation details. However, in this case, the only benefit is to satisfy your desire to keep `the-environment' out of psyntax. This conservatism is preventing Guile from adding features. And we do need features -- local-eval and syntax-parse among them. For the record, I absolutely want Stefan to have what he needs to implement `syntax-parse'. My understanding is that all he needs is a way to associate properties with syntactic keywords. Toward that end, I proposed that we should add something analogous to procedure properties for macros. This can be done without exposing internal details as you have done. But what if they're the wrong interface? Well, then we use the normal deprecation mechanism to get rid of them, eventually (in the 2.2 series, for example). We learned something. Users get warned off the code. Life goes on. Yes, this is always an option, but it is a _painful_ option for all involved. If we can already foresee the need to deprecate an interface, wouldn't it be better not to add it in the first place? * * * * * Anyway, I can plainly see that your mind is set on this, and that all of the above words were a waste of effort, so I'm going to try to drop it. I just have one final request: please at least change the lexical environments in your `local-eval' implementation to use the future-proof `evaluator procedure' representation, as I have done in mine. FWIW, criticisms aside, I _do_ very much appreciate that you heard my plea for local-eval in 2.0.4, and that you have put so much time into this. Thanks for that, and for all the wonderful things you have done for Guile and GNU. Mark
Re: syntax-local-binding
Hi, Mark H Weaver m...@netris.org skribis: Because it breaks your nice equivalence. For example: (let ((x 1)) (syntax-local-binding #'x)) is not equivalent to: (let ((x 1)) (local-eval '(syntax-local-binding #'x) (the-environment))) Put another way: if anyone uses `syntax-local-binding' to distinguish lexical variables from macros in some clever macro of theirs, this means that `local-eval' is now buggy with regard to their clever macro. What about recommending against “clever macros” that use ‘syntax-local-binding’, or documenting the limitation in how ‘local-eval’ and ‘syntax-local-binding’ would interact? After all, the point of ‘local-eval’ is to provide a compatibility later with 1.8, and ‘syntax-local-binding’ didn’t exist there. Thanks, Ludo’.
Re: syntax-local-binding
Andy Wingo wi...@pobox.com writes: `lexical' A lexically-bound variable. The value is a unique token (in the sense of `eq?') identifying this binding. `macro' A syntax transformer, either local or global. The value is the transformer procedure. Ironically, `syntax-local-binding' renders the current simple implementation strategy of `the-environment' inadequate, because identifier-syntax is no longer sufficient to simulate a lexical. More importantly, this exposing of internal binding representations will interfere with our ability to change the representations later. In particular, I was hoping to change the binding representation of `syntax-rules' macros so that they are serializable. In particular, they would be represented by the `syntax-rules' form itself (the same one that psyntax currently passes to `primitive-eval' to produce the transformer procedure). A weak-key hash table would cache the compiled transformer procedures. This would allow (the-environment) to capture locally-bound `syntax-rules' macros. Unfortunately, `syntax-local-binding', as currently documented, now makes this impossible. Thanks, Mark
Re: syntax-local-binding
On Fri 20 Jan 2012 21:26, Mark H Weaver m...@netris.org writes: Andy Wingo wi...@pobox.com writes: `lexical' A lexically-bound variable. The value is a unique token (in the sense of `eq?') identifying this binding. `macro' A syntax transformer, either local or global. The value is the transformer procedure. Ironically, `syntax-local-binding' renders the current simple implementation strategy of `the-environment' inadequate, because identifier-syntax is no longer sufficient to simulate a lexical. Why do you say that? In particular, I was hoping to change the binding representation of `syntax-rules' macros so that they are serializable. In particular, they would be represented by the `syntax-rules' form itself (the same one that psyntax currently passes to `primitive-eval' to produce the transformer procedure). A weak-key hash table would cache the compiled transformer procedures. This would allow (the-environment) to capture locally-bound `syntax-rules' macros. Unfortunately, `syntax-local-binding', as currently documented, now makes this impossible. Why do you think that? The procedures do carry metadata; I understood that that was your strategy, to use the serialization of the syntax-rules form in the procedure metadata. Andy -- http://wingolog.org/
Re: syntax-local-binding
Andy Wingo wi...@pobox.com writes: On Fri 20 Jan 2012 21:26, Mark H Weaver m...@netris.org writes: Andy Wingo wi...@pobox.com writes: `lexical' A lexically-bound variable. The value is a unique token (in the sense of `eq?') identifying this binding. `macro' A syntax transformer, either local or global. The value is the transformer procedure. Ironically, `syntax-local-binding' renders the current simple implementation strategy of `the-environment' inadequate, because identifier-syntax is no longer sufficient to simulate a lexical. Why do you say that? Because it breaks your nice equivalence. For example: (let ((x 1)) (syntax-local-binding #'x)) is not equivalent to: (let ((x 1)) (local-eval '(syntax-local-binding #'x) (the-environment))) Put another way: if anyone uses `syntax-local-binding' to distinguish lexical variables from macros in some clever macro of theirs, this means that `local-eval' is now buggy with regard to their clever macro. in particular, I was hoping to change the binding representation of `syntax-rules' macros so that they are serializable. In particular, they would be represented by the `syntax-rules' form itself (the same one that psyntax currently passes to `primitive-eval' to produce the transformer procedure). A weak-key hash table would cache the compiled transformer procedures. This would allow (the-environment) to capture locally-bound `syntax-rules' macros. Unfortunately, `syntax-local-binding', as currently documented, now makes this impossible. Why do you think that? The procedures do carry metadata; I understood that that was your strategy, to use the serialization of the syntax-rules form in the procedure metadata. Well, this was in the context of a new strategy where psyntax would include a new core form called `call-with-current-local-expander' that calls its parameter (a procedure or macro) with a procedure that accepts an expression and returns an expanded form. In this case, the most straightforward implementation would simply serialize the (r w mod) structures directly. Toward that end, I was thinking it would be nice to keep those structures serializable. The only part that's not currently serializable are the transformer procedures for local macros. Thus the change in representation. More specifically: instead of attaching procedure properties/metadata to the transformer procedures as is currently done, that information would be put directly into the binding. An alist would probably make the most sense. The source code of the macro (whatever's currently passed to `primitive-eval') would also be included. This may or may not be a good idea, I dunno. I haven't fully thought it through. However, that's not the point. The point is that by exposing these details in the API, you are constraining our ability to make changes of this kind. I think that's very unfortunate. Mark
Re: local-eval on syntax-local-binding, bound-identifiers
On Tue 17 Jan 2012 00:27, Andy Wingo wi...@pobox.com writes: TBH I think this is the best thing we can do for local-eval. We preserve flexibility for local-eval, make other experiments possible, and the local-eval implementation is a bit more perspicacious, as the scoping is more lexical (in the same file, even). Perspicuous, rather. How embarrassing! Andy -- http://wingolog.org/
Re: syntax-local-binding
Hello, I have now pushed an implementation of syntax-local-binding to stable-2.0, with the following documentation. In the spirit of Eli's note on Racket's syntax-local-value, it also works with identifiers that are bound at the module level or the top level. Comments and patches welcome. Cheers, Andy -- Scheme Procedure: syntax-local-binding id Resolve the identifer ID, a syntax object, within the current lexical environment, and return two values, the binding type and a binding value. The binding type is a symbol, which may be one of the following: `lexical' A lexically-bound variable. The value is a unique token (in the sense of `eq?') identifying this binding. `macro' A syntax transformer, either local or global. The value is the transformer procedure. `pattern-variable' A pattern variable, bound via syntax-case. The value is an opaque object, internal to the expander. `displaced-lexical' A lexical variable that has gone out of scope. This can happen if a badly-written procedural macro saves a syntax object, then attempts to introduce it in a context in which it is unbound. The value is `#f'. `global' A global binding. The value is a pair, whose head is the symbol, and whose tail is the name of the module in which to resolve the symbol. `other' Some other binding, like `lambda' or other core bindings. The value is `#f'. This is a very low-level procedure, with limited uses. One case in which it is useful is to build abstractions that associate auxiliary information with macros: (define aux-property (make-object-property)) (define-syntax-rule (with-aux aux value) (let ((trans value)) (set! (aux-property trans) aux) trans))) (define-syntax retrieve-aux (lambda (x) (syntax-case x () ((x id) (call-with-values (lambda () (syntax-local-binding #'id)) (lambda (type val) (with-syntax ((aux (datum-syntax #'here (and (eq? type 'macro) (aux-property val) #''aux))) (define-syntax foo (with-aux 'bar (syntax-rules () ((_) 'foo (foo) = foo (retrieve-aux foo) = bar `syntax-local-binding' must be called within the dynamic extent of a syntax transformer; to call it otherwise will signal an error. -- http://wingolog.org/
Re: local-eval on syntax-local-binding, bound-identifiers
Andy Wingo wi...@pobox.com writes: What if instead we implemented closure serialization somehow? Then we would handle procedural macros too, and bound-identifiers would still be sufficient. Maybe that idea is a little too crazy. Are we still talking about Scheme? The language with call-with-current-continuation? a little too crazy is not a criterion. Too complex to work or or with would be. Those are related, but not necessarily the same. -- David Kastrup
Re: local-eval on syntax-local-binding, bound-identifiers
Hi Andy! Andy Wingo wi...@pobox.com writes: + (cons (wrap (car symnames) + (anti-mark (make-wrap (car marks) subst)) * Why are you adding anti-marks here? As the changelog noted (and a comment should have noted ;), the identifiers are anti-marked so that syntax transformers can introduce them, as-is. The purpose of this procedure is to get a list of identifiers, and to capture some subset of them. It will do so by introducing references to them in the expansion of some macro. However they are not introduced identifiers: they come from the code itself. They are input the macro, and as such need an anti-mark. The anti-mark will be stripped from the expansion when the transformer that called `bound-identifiers' returns. Does this mean that `bound-identifiers' will not function properly when used outside of a macro? What about if it's used within a macro that was generated by another macro (or things of that nature)? Are there cases where you might need to strip more than one anti-mark? To use your phrase, this has a bad smell. More importantly: I notice that you are not stripping the psyntax wrap from identifiers placed within the wrapper procedure above. There are certainly benefits to that, but remember that the wrapper procedure will in general be serialized to disk and evaluated in a different Guile session, where the gensym counters have been reset. Of course, like all macros! The forgeable gensym issue is something we have in Guile, more generally, that needs a broader solution. Ah, good point! Macros already serialize syntax-objects to disk. psyntax wraps are already part of our ABI, so nothing new there. However, I fear that the gensym issue might be a serious problem for `local-eval', even though it hasn't been a problem for macros. The reason it has not been a problem with macros is that, within a top-level macro (which are the only ones used across Guile sessions), the only syntax-objects that can be meaningfully _introduced_ into the expansion are top-level/module bindings. But these bindings have no associated labels or gensyms, because they're not in the wrap. On the other hand, with `local-eval', it seems to me quite plausible that gensym collisions might occur. Suppose in one Guile session you compile a procedure (foo) that uses (the-environment), and then in another Guile session, you call (foo) and then `local-eval' with the environment returned by (foo). Now the wrapper procedure splices together syntax objects from two different Guile sessions into a single top-level form, where (unlike in the macro case) all of these syntax objects are lexicals, and thus depend on the gensyms and the labels. See how this is a problem now where it wasn't before? Or am I missing something? +((module? e) + ;; Here we evaluate the expression within `lambda', and then + ;; call the resulting procedure outside of the dynamic extent + ;; of `eval'. We do this because `eval' sets (current-module) + ;; within its dynamic extent, and we don't want that. Also, + ;; doing it this way makes this a proper tail call. + ((eval #`(lambda () #,x) e))) * This was my mistake, but since I'm already marking up the code: the `lambda' wrap above needs a `#f' before `e' to force expression context. OK. (Note though that (eval X e) does indeed evaluate X in tail position.) Looks to me like `eval' is initially bound to the C function scm_eval. Is it later rebound to a Scheme procedure? If so, where? For the record, I still think it's better for `the-environment' to be implemented within psyntax as a core form. It's a fundamental syntactic construct with clean semantics, and it belongs in psyntax with its brethren. Your desire to remove it from psyntax has caused you to add far less elegant interfaces that have been hastily designed, and that may not even be sufficient for a full implementation of `the-environment' that captures mutually-recursive local macros. In pursuit of the goal of agreeing on a strategy, I would like to convince you that you are wrong on all of these points :) So, in that spirit, I argue: Very well, I will endeavor to be open-minded. `the-environment' is not fundamental: it can be implemented in terms of simpler primitives. The same can be said of `lambda' or `syntax-case', but that's not the appropriate way to choose primitives in a language. The set of primitives chosen in Scheme are not the ones that are simplest to implement. It makes more sense to choose primitives with simple, clean semantics, which segues nicely into your next paragraph. `the-environment' does not have clean semantics, inasmuch as it has nothing worthy of the name, not yet anyway. The lambda calculus, the scheme language, even the syntax-case system have well-studied semantics (denotational and/or operational), and lots of
Re: local-eval on syntax-local-binding, bound-identifiers
Hi Andy! Thanks again for working on this. Andy Wingo wi...@pobox.com writes: * Why are you adding anti-marks here? As the changelog noted (and a comment should have noted ;), the identifiers are anti-marked so that syntax transformers can introduce them, as-is. The purpose of this procedure is to get a list of identifiers, and to capture some subset of them. It will do so by introducing references to them in the expansion of some macro. However they are not introduced identifiers: they come from the code itself. They are input the macro, and as such need an anti-mark. The anti-mark will be stripped from the expansion when the transformer that called `bound-identifiers' returns. Does this mean that `bound-identifiers' will not function properly when used outside of a macro? What about if it's used within a macro that was generated by another macro (or things of that nature)? Are there cases where you might need to strip more than one anti-mark? Well, bound-identifiers is a procedure, so if you are using it outside the dynamic extent of a transformer procedure, that means that you have a syntax object that you squirreled away from somewhere, so already we're in somewhat uncharted territory. How about something like (bound-identifiers #'here)? or (bound-identifiers #'x) where `x' is some lexical variable? Macro-generating macros should be fine, here. `expand-macro' is iterative, not recursive, so you don't need to strip anti-marks twice. Ah, okay. Good point! I agree that this anti-mark has a bad smell, but the idea of a `bound-identifiers' procedure or form sounds like a good idea, so if you have any suggestions for improvement here, they are most welcome. As I've already said, I don't think `bound-identifiers' will be useful in a full implementation of `local-eval', so once we move to that improved implementation, `bound-identifiers' will be left around as an orphan: a primitive of dubious value, introduced specifically to implement something that it turned out to be insufficient for. If you insist on this strategy, I think what we really need is a list of ribs, where each rib also specifies whether it is recursive. We don't actually care about `let' vs `letrec' (though there's no harm in providing that information in the interface, and it probably makes sense to for consistency), but we _do_ care about the difference between `let-syntax', `letrec-syntax', and internal bodies with mutually-recursive `define-syntax' forms. See how we're exposing increasingly complex internal psyntax structures in order to achieve your dream of making `local-eval' sleep outside in the shed? [W]ith `local-eval', it seems to me quite plausible that gensym collisions might occur. Suppose in one Guile session you compile a procedure (foo) that uses (the-environment), and then in another Guile session, you call (foo) and then `local-eval' with the environment returned by (foo). Now the wrapper procedure splices together syntax objects from two different Guile sessions into a single top-level form, where (unlike in the macro case) all of these syntax objects are lexicals, and thus depend on the gensyms and the labels. See how this is a problem now where it wasn't before? Or am I missing something? To be perfectly honest, this stuff is very confusing to me, but I think I can see how this can happen, yes. I do think that it's important to fix this bug at some point, but IMO it is not a blocker for local-eval, much less 2.0.4. I strongly disagree. Your implementation will clearly be buggy without a proper solution to the collision of gensyms (labels and marks, at least). I don't know about you, but personally I prefer rock-solid code with clearly documented limitations (that almost no one is likely to hit anyway) to buggy code. If you don't want to deal with the gensym problem for 2.0.4, there's an easy solution. Simply strip the wraps for now (as is done by my patch), and everything will robust as long as we don't capture local syntax. BTW, did you see my most recent model for thinking about `local-eval'? (the-environment) expands to (list (lambda () expr) ...), with one element for every possible expression: a countably infinite list that could be built lazily. `local-eval' simply chooses the appropriate procedure from the list and calls it. A poor implementation strategy, but the semantic meaning is quite clear, no? It sounds clear, but does it have any explanatory power? It sounds like it could apply just as well to any other computation... I don't understand what you mean here. It seems to me that this model can answer any question you could possibly have about the observable behaviors of `the-environment' and `local-eval', besides their efficiency. Can you provide a counter-example to this claim? Creating wraps is not the hack. It's creating wraps that are scoped in another specific module. With the-environment in psyntax, psyntax
Re: local-eval on syntax-local-binding, bound-identifiers
Hi Mark, On Mon 16 Jan 2012 21:36, Mark H Weaver m...@netris.org writes: Thanks again for working on this. And thank you again for all your work, and patience with my pigheadedness. if you insist in this foolish quest to banish `the-environment' to sleep in the shed as a second-class citizen, I cannot stop you :) TBH I think this is the best thing we can do for local-eval. We preserve flexibility for local-eval, make other experiments possible, and the local-eval implementation is a bit more perspicacious, as the scoping is more lexical (in the same file, even). I know there's a smilie in your statement, but really, it's not just local-eval: there's loads more that should be broken out into modules over time, somehow :) Think of it as building a hippie commune of functionality, instead of making everyone live in the same house :) (OK, that's stretching it a bit, but perhaps it is partially apt?) Now, specific commentary. How about something like (bound-identifiers #'here)? scheme@(guile-user) (bound-identifiers #'here) $5 = () scheme@(guile-user) (let ((x 10)) (bound-identifiers #'here)) $6 = (#(syntax-object x ((#f top) shift #(ribcage #(x) #((top)) #(i176))) (hygiene guile-user))) What should the answer be in this case? Would you expect `x' in the list? Certainly for the-environment you would. But here: scheme@(guile-user) (define-syntax bound-here (lambda (x) (with-syntax (((id ...) (map (lambda (id) (datum-syntax x id)) (bound-identifiers #'here #'(list 'id ... scheme@(guile-user) bound-here $7 = (#(syntax-object x ((#f top) shift #(ribcage #(x) #((top)) #(i192))) (hygiene guile-user))) scheme@(guile-user) (let ((y 10)) bound-here) $8 = (#(syntax-object x ((#f top) shift #(ribcage #(x) #((top)) #(i192))) (hygiene guile-user))) So, it seems to be sensible. Now, what to do with these identifiers: you if you introduce one into another macro, the mark will indeed be stripped. I'm not sure what else you can do with a syntax-object, actually! Pass it directly to eval or compile, I guess, and in that case we do lose, as the anti-mark isn't stripped. But that's the case for other syntax objects captured in a syntax transformer, as well. Should we anti-mark only within the dynamic extent of a transformer, I wonder? As I've already said, I don't think `bound-identifiers' will be useful in a full implementation of `local-eval', so once we move to that improved implementation, `bound-identifiers' will be left around as an orphan: a primitive of dubious value, introduced specifically to implement something that it turned out to be insufficient for. Hum. Definitely something to think about. What if instead we implemented closure serialization somehow? Then we would handle procedural macros too, and bound-identifiers would still be sufficient. Maybe that idea is a little too crazy. If we have to lexical contours associated with bindings, recursive is only one bit: you probably also need letrec vs letrec*. To be perfectly honest, this stuff is very confusing to me, but I think I can see how this can happen, yes. I do think that it's important to fix this bug at some point, but IMO it is not a blocker for local-eval, much less 2.0.4. I strongly disagree. Your implementation will clearly be buggy without a proper solution to the collision of gensyms (labels and marks, at least). I don't know about you, but personally I prefer rock-solid code with clearly documented limitations (that almost no one is likely to hit anyway) to buggy code. If you don't want to deal with the gensym problem for 2.0.4, there's an easy solution. Simply strip the wraps for now (as is done by my patch), and everything will robust as long as we don't capture local syntax. Thinking about it a little more, labels are a non-issue. All they need to be is unique in the sense of eq?. Labels are strings. If they are loaded in separate compilation units, they will be unique, no matter what their contents. Labels are more important than marks, also, for the correctness of the algorithm. A mark collision is only an issue if there is also a symbolic collision. Label collision could alias completely unrelated bindings. Anyway, I would rather serialize bad marks than no marks. That's my personal opinion ;-) But if you think this is a huge issue, let's fix the marks to be more unique, no? Note that there is a well-known optimization that you don't actually need to generate the characters corresponding to a gensym until they are needed. It might serve your purposes. OK, I'm getting very sleepy now :) Let me know your thoughts. It would be great if all of this could land before Sunday. Though the cricket folk say pace is nothing without guile, Guile is nothing without a good development pace ;-) Cheers, Andy --
syntax-local-binding
Hi list, Attached is a patch that implements a new accessor, syntax-local-binding. It's like syntax-local-value, but can also determine if an identifier is a pattern var or a normal lexical. (define-syntax local-binding (lambda (x) (syntax-case x () ((_ id) (identifier? #'id) (call-with-values (lambda () (syntax-local-binding #'id)) (lambda (x y) (with-syntax ((type (datum-syntax #'here x))) #''type))) (let-syntax ((x (lambda (x) 10))) (local-binding x)) $3 = local-macro (syntax-case #'(foo) () ((id) (local-binding id))) $4 = pattern-variable (let ((x 10)) (local-binding x)) $5 = lexical It returns two values. The first is a symbol, either `local-macro', `lexical', or `pattern-variable'. The second is specific to the type. For local-macro it is the transformer binding. For lexical it is the gensym name of the lexical. For pattern variables, it is something opaque that identifies that pattern var. Inspired by a patch by Stefan Israelsson Tampe. I'm going to try to implement the-environment with this. We'll see how it goes! Andy -- http://wingolog.org/
Re: syntax-local-binding
On Sun 15 Jan 2012 18:00, Andy Wingo wi...@pobox.com writes: Attached is a patch that implements a new accessor, syntax-local-binding. Here 'tis! From 09ba44abeb47cdf4ec61df6f7386217f0cbe30c7 Mon Sep 17 00:00:00 2001 From: Andy Wingo wi...@pobox.com Date: Sun, 15 Jan 2012 17:51:02 +0100 Subject: [PATCH] add syntax-local-binding * module/ice-9/boot-9.scm (syntax-local-binding): New binding. * module/ice-9/psyntax.scm: Locally define a fluid that holds the transformer environment. with-transformer-environment calls a procedure with the transformer environment, or raises an error if called outside the extent of a transformer. Bind transformer-environment in expand-macro. (syntax-local-binding): New procedure to return binding information of a lexially bound identifier (a lexical, local macro, or pattern variable). --- module/ice-9/boot-9.scm |1 + module/ice-9/psyntax.scm | 39 +-- 2 files changed, 38 insertions(+), 2 deletions(-) diff --git a/module/ice-9/boot-9.scm b/module/ice-9/boot-9.scm index f661d08..9cdd8d1 100644 --- a/module/ice-9/boot-9.scm +++ b/module/ice-9/boot-9.scm @@ -389,6 +389,7 @@ If there is no handler at all, Guile prints an error and then exits. (define generate-temporaries #f) (define bound-identifier=? #f) (define free-identifier=? #f) +(define syntax-local-binding #f) ;; $sc-dispatch is an implementation detail of psyntax. It is used by ;; expanded macros, to dispatch an input against a set of patterns. diff --git a/module/ice-9/psyntax.scm b/module/ice-9/psyntax.scm index 1bf3c32..dcabafe 100644 --- a/module/ice-9/psyntax.scm +++ b/module/ice-9/psyntax.scm @@ -786,6 +786,14 @@ id)) (else (syntax-violation 'id-var-name invalid id id) +(define transformer-environment + (make-fluid + (lambda (k) + (error called outside the dynamic extent of a syntax transformer + +(define (with-transformer-environment k) + ((fluid-ref transformer-environment) k)) + ;; free-id=? must be passed fully wrapped ids since (free-id=? x y) ;; may be true even if (free-id=? (wrap x w) (wrap y w)) is not. @@ -1321,8 +1329,10 @@ (syntax-violation #f encountered raw symbol in macro output (source-wrap e w (wrap-subst w) mod) x)) (else (decorate-source x s) -(rebuild-macro-output (p (source-wrap e (anti-mark w) s mod)) - (new-mark +(with-fluids ((transformer-environment + (lambda (k) (k e r w s rib mod + (rebuild-macro-output (p (source-wrap e (anti-mark w) s mod)) +(new-mark) (define expand-body ;; In processing the forms of the body, we create a new, empty wrap. @@ -2435,6 +2445,31 @@ (set! syntax-source (lambda (x) (source-annotation x))) +(set! syntax-local-binding + (lambda (id) +(arg-check nonsymbol-id? id 'syntax-local-value) +(with-transformer-environment + (lambda (e r w s rib mod) + (define (strip-anti-mark w) + (let ((ms (wrap-marks w)) (s (wrap-subst w))) + (if (and (pair? ms) (eq? (car ms) the-anti-mark)) + ;; output is from original text + (make-wrap (cdr ms) (if rib (cons rib (cdr s)) (cdr s))) + ;; output introduced by macro + (error what!!! + (let ((label (id-var-name (syntax-object-expression id) + (strip-anti-mark (syntax-object-wrap id) + (if (not (string? label)) + (error identifier not lexically bound id)) + (let ((b (assq-ref r label))) + (if (not b) + (error displaced lexical id)) + (case (binding-type b) + ((lexical) (values 'lexical (binding-value b))) + ((macro) (values 'local-macro (binding-value b))) + ((syntax) (values 'pattern-variable (binding-value b))) + (else (error unpossible! b) + (set! generate-temporaries (lambda (ls) (arg-check list? ls 'generate-temporaries) -- 1.7.8.3 -- http://wingolog.org/
local-eval on syntax-local-binding, bound-identifiers
Hi Mark, I had made some noise about preferring an implementation of local-eval based on primitives from psyntax. But, I didn't clarify my argument by providing the primitives. Here are some patches that provide syntax-local-binding, as I noted in my previous mail, and also a procedure to get all identifiers that are visible within the scope of another identifier. I then refactored your patch to implement local-eval entirely in terms of these primitives and other normal macrology. What do you think? In the interests of time and debugging, I removed the support for pattern variables; it should be easy to add back. These are preliminary patches, but if this approach proves to be viable, I would prefer it to one that bakes the-environment into psyntax. WDYT? Regards, Andy From 48d8e52e316984f2bf9380df85079bb5fa142253 Mon Sep 17 00:00:00 2001 From: Andy Wingo wi...@pobox.com Date: Sun, 15 Jan 2012 17:51:02 +0100 Subject: [PATCH 1/3] add syntax-local-binding * module/ice-9/boot-9.scm (syntax-local-binding): New binding. * module/ice-9/psyntax.scm: Locally define a fluid that holds the transformer environment. with-transformer-environment calls a procedure with the transformer environment, or raises an error if called outside the extent of a transformer. Bind transformer-environment in expand-macro. (syntax-local-binding): New procedure to return binding information of a lexically bound identifier (a lexical, local macro, a pattern variable, or a displaced lexical). --- module/ice-9/boot-9.scm |1 + module/ice-9/psyntax.scm | 39 +-- 2 files changed, 38 insertions(+), 2 deletions(-) diff --git a/module/ice-9/boot-9.scm b/module/ice-9/boot-9.scm index f661d08..9cdd8d1 100644 --- a/module/ice-9/boot-9.scm +++ b/module/ice-9/boot-9.scm @@ -389,6 +389,7 @@ If there is no handler at all, Guile prints an error and then exits. (define generate-temporaries #f) (define bound-identifier=? #f) (define free-identifier=? #f) +(define syntax-local-binding #f) ;; $sc-dispatch is an implementation detail of psyntax. It is used by ;; expanded macros, to dispatch an input against a set of patterns. diff --git a/module/ice-9/psyntax.scm b/module/ice-9/psyntax.scm index 1bf3c32..30685bc 100644 --- a/module/ice-9/psyntax.scm +++ b/module/ice-9/psyntax.scm @@ -786,6 +786,14 @@ id)) (else (syntax-violation 'id-var-name invalid id id) +(define transformer-environment + (make-fluid + (lambda (k) + (error called outside the dynamic extent of a syntax transformer + +(define (with-transformer-environment k) + ((fluid-ref transformer-environment) k)) + ;; free-id=? must be passed fully wrapped ids since (free-id=? x y) ;; may be true even if (free-id=? (wrap x w) (wrap y w)) is not. @@ -1321,8 +1329,10 @@ (syntax-violation #f encountered raw symbol in macro output (source-wrap e w (wrap-subst w) mod) x)) (else (decorate-source x s) -(rebuild-macro-output (p (source-wrap e (anti-mark w) s mod)) - (new-mark +(with-fluids ((transformer-environment + (lambda (k) (k e r w s rib mod + (rebuild-macro-output (p (source-wrap e (anti-mark w) s mod)) +(new-mark) (define expand-body ;; In processing the forms of the body, we create a new, empty wrap. @@ -2435,6 +2445,31 @@ (set! syntax-source (lambda (x) (source-annotation x))) +(set! syntax-local-binding + (lambda (id) +(arg-check nonsymbol-id? id 'syntax-local-value) +(with-transformer-environment + (lambda (e r w s rib mod) + (define (strip-anti-mark w) + (let ((ms (wrap-marks w)) (s (wrap-subst w))) + (if (and (pair? ms) (eq? (car ms) the-anti-mark)) + ;; output is from original text + (make-wrap (cdr ms) (if rib (cons rib (cdr s)) (cdr s))) + ;; output introduced by macro + (make-wrap ms (if rib (cons rib s) s) + (let ((label (id-var-name (syntax-object-expression id) + (strip-anti-mark (syntax-object-wrap id) + (if (not (string? label)) + (error identifier not lexically bound id)) + (let ((b (assq-ref r label))) + (if b + (case (binding-type b) + ((lexical) (values 'lexical (binding-value b))) + ((macro) (values 'local-macro (binding-value b))) + ((syntax) (values 'pattern-variable (binding-value b))) + (else (error unpossible! b))) + (values
Re: local-eval on syntax-local-binding, bound-identifiers
Hi Andy, Thanks very much for heeding my call for `local-eval' in 2.0.4, and for putting so much time into this. For the record, I still think it's better for `the-environment' to be implemented within psyntax as a core form. It's a fundamental syntactic construct with clean semantics, and it belongs in psyntax with its brethren. Your desire to remove it from psyntax has caused you to add far less elegant interfaces that have been hastily designed, and that may not even be sufficient for a full implementation of `the-environment' that captures mutually-recursive local macros. That said, there's a lot to like in your implementation, and it has some notable improvements over mine. It also has some problems, not all of which are trivial. Please see below, where I have inserted specific questions and comments into your patch. Thanks again, Mark [... skipped the first patch, which looks very well implemented, though I'm still not sure that we should be exposing this in our API ...] From 2c3da44320019453115811af386febaa7eb241c3 Mon Sep 17 00:00:00 2001 From: Andy Wingo wi...@pobox.com Date: Sun, 15 Jan 2012 18:39:44 +0100 Subject: [PATCH 2/3] add bound-identifiers * module/ice-9/boot-9.scm (bound-identifiers): Declare variable. * module/ice-9/psyntax.scm: Add all-bound-identifiers helper, and define bound-identifiers. The identifiers are anti-marked so that syntax transformers can introduce them, as-is. --- module/ice-9/boot-9.scm |1 + module/ice-9/psyntax.scm | 49 ++ 2 files changed, 50 insertions(+), 0 deletions(-) diff --git a/module/ice-9/boot-9.scm b/module/ice-9/boot-9.scm index 9cdd8d1..b8aa842 100644 --- a/module/ice-9/boot-9.scm +++ b/module/ice-9/boot-9.scm @@ -389,6 +389,7 @@ If there is no handler at all, Guile prints an error and then exits. (define generate-temporaries #f) (define bound-identifier=? #f) (define free-identifier=? #f) +(define bound-identifiers #f) (define syntax-local-binding #f) ;; $sc-dispatch is an implementation detail of psyntax. It is used by diff --git a/module/ice-9/psyntax.scm b/module/ice-9/psyntax.scm index 30685bc..25543e0 100644 --- a/module/ice-9/psyntax.scm +++ b/module/ice-9/psyntax.scm @@ -786,6 +786,48 @@ id)) (else (syntax-violation 'id-var-name invalid id id) +;; +;; all-bound-identifiers returns a list of all lexically bound +;; identifiers, as syntax objects. They are in order from outer to +;; inner. +;; +(define all-bound-identifiers + (lambda (w mod) +(define scan + (lambda (subst results) +(if (null? subst) +results +(let ((fst (car subst))) + (if (eq? fst 'shift) + (scan (cdr subst) results) + (let ((symnames (ribcage-symnames fst)) +(marks (ribcage-marks fst))) +(if (vector? symnames) +(scan-vector-rib subst symnames marks results) +(scan-list-rib subst symnames marks results +(define scan-list-rib + (lambda (subst symnames marks results) +(let f ((symnames symnames) (marks marks) (results results)) + (if (null? symnames) + (scan (cdr subst) results) + (f (cdr symnames) (cdr marks) + (cons (wrap (car symnames) + (anti-mark (make-wrap (car marks) subst)) * Why are you adding anti-marks here? + mod) + results)) +(define scan-vector-rib + (lambda (subst symnames marks results) +(let ((n (vector-length symnames))) + (let f ((i 0) (results results)) +(if (fx= i n) +(scan (cdr subst) results) +(f (fx+ i 1) + (cons (wrap (vector-ref symnames i) + (anti-mark (make-wrap (vector-ref marks i) subst)) * Ditto. + mod) + results))) +(scan (wrap-subst w) '( + (define transformer-environment (make-fluid (lambda (k) @@ -2470,6 +2512,13 @@ (else (error unpossible! b))) (values 'displaced-lexical #f +(set! bound-identifiers + (lambda (x) +(arg-check nonsymbol-id? x 'bound-identifiers) +(reverse + (all-bound-identifiers (syntax-object-wrap x) +(syntax-object-module x) + (set! generate-temporaries (lambda (ls) (arg-check list? ls 'generate-temporaries
