Re: [racket-users] syntax-property lost across module boundary (WAS: format-id doesn't preserve preserved?-edness of syntax-property?)
All, Thanks for help debugging and explaining this. FYI, with Matthew's bug fix merged, and help from Alexis, I've managed to work around my issue. It was pretty much the same issue described in thread linked to by Dupéron, https://groups.google.com/forum/#!topic/racket-users/TGax2h8dVxs, and also in the Racket GitHub issue https://github.com/racket/racket/issues/1495. The workaround seems to be: if you want to stash an identifier in a syntax-property, expand into syntax that attaches the syntax-property. That is, don't do (define x (syntax-property #'id 'prop #'stx)) and then use x in a syntax object, but generate #`(... (syntax-property #'id 'prop #'stx ...)) In my example, the identifier that caused issues was generated by define^, which directly called syntax-property in a for-syntax helper function. The identifier that worked fine was generated by define^^, which generated syntax that included a syntax-property. Maybe we shouldn't be stashing identifiers in syntax-properties at all, but in practice this workaround seems to work (in the implementations of Alexis' Rascal and my Cur). -- William J. Bowman On Wed, Dec 14, 2016 at 08:14:49AM -0700, Matthew Flatt wrote: > I think there are two problems in this example. > > > The first problem is that preserved syntax properties were not actually > preserved in bytecode for a syntax object that has no source location. > (I'm assuming that Alexis's tests were run in DrRacket in it's default > mode of compiling bytecode for `require`d files.) I'll push a repair > for that bug. > > > The second problem is that syntax objects as property values don't work > the way that you want. That limitation is why, even with the > property-presevation bug fixed, the `free-identifier=?` test fails. > Unfortunately, that's a limitation of syntax objects and properties > that's difficult to resolve. > > To explain a little more, consider the program > > https://gist.github.com/mflatt/d569d8f3a89e6815cf18df26adc11053 > > For now, look at just the `property` submodule. The > `quote-syntax/add-example-property` macro is like `quote-syntax`, > except that it adds an 'example property that holds the current time at > compile time. > > An expression like > > (syntax-property (quote-syntax/add-example-property id) 'example) > > accesses that compile-time value at run time. If you compile the > program to bytecode, the value shown by running the module later still > reports the time that the module was compiled. > > Now consider the enclosing module, which uses the `property` submodule > and accesses the time for both a freshly quoted term and the value of > `id` as exported by the submodule: > > (require 'property) > (syntax-property (quote-syntax/add-example-property new-id) 'example) > (syntax-property id 'example) > > I think you won't be too surprised that the freshly quoted identifier > has a newer time than the one reported for `id`. Even though both > values were obtained from the same `compile-time` variable, each module > compilation (submodule and enclosing module) gets a fresh instance of > `compile-time`. > > > In "format-id1.rkt", instead of `compile-time` bound to a timestamp, > you have a quoted syntax object that you can image being implicitly > lifted as > > (define-for-syntax compile-time #'id^) > > In much the same way that `(current-inexact-milliseconds)` in > > (define-for-syntax compile-time (current-inexact-milliseconds)) > > produces a different result for each subsequent compile-time use of the > modules, `#'id^` produces a different syntax object depending on how > its enclosing module is instantiated. Imagine taking the compiled > bytecode, saving it in two places, and loading it as a module from each > place; the `#'id` evaluations for the copies will produce identifiers > that claim to be from different modules relative to the filesystem. > > So, here's the limitation: the implicit lifting that happens for a > quoted syntax object, which enables a kind of re-evaluation of that > syntax object in each module instance, doesn't happen for > syntax-property values. More generally, changes to the scope of a > syntax object are not propagated to property values that happen to be > themselves syntax objects. When you put a syntax object into a > property, then you get whatever that syntax object meant at the time it > was attached as a property. > > That might still sound ok for your case, because you compile and use > "format-id1.rkt" in-place. Still, the information attached to `#'id^` > changes. While compiling the module, the attached binding information > says "the current module being compiled". When compiling > "format-id2.rkt", the attached binding information for that `#'id^` > says "format-id1.rkt". > > > To summarize, don't try to attach syntax objects as property values > like that. I know this advice sounds ironic, given that the original > use of properties was for syntax-valued keys like 'origin.
Re: [racket-users] syntax-property lost across module boundary (WAS: format-id doesn't preserve preserved?-edness of syntax-property?)
> What are the implications of this for the Types as Macros/Turnstile stuff? As William mentioned, he's also working on a language using the Types as Macros approach. Alex and I have run into the same issue, but it's always with compiled files (without compiling, requires and provides appear to behave as expected). Unfortunately, we have not figured out a good solution either. Could we borrow any ideas from Typed Racket here? > but I’d like to know if I’m relying on some sort of undefined behavior In some sense, the entire Types as Macros approach is "undefined behavior" :/ because we are using the expander for unintended purposes. So thank you (and William and everyone else that has tried it out) for diving in and helping us figure out what works and what needs more work. But now that we've shown that such an approach is both viable and wanted by Racket programmers, for the rough parts where we cannot find a good solution, the next step might be to address these issues with contributions to the expander itself. Just to throw them out there, other issues that I'm working on (in addition to scopes on stx props) are: - Better support (over stop lists) for "delimited" expansion. For example, I often want to "fully expand an expression, but not past a specified language". - Better support for expanding expressions with free identifiers in a context. - Better support for mapping stx objs back to the surface stx, in a hygienic way. On Wed, Dec 14, 2016 at 6:50 PM, Alexis King wrote: >> On Dec 14, 2016, at 7:14 AM, Matthew Flatt wrote: >> >> To summarize, don't try to attach syntax objects as property values >> like that. I know this advice sounds ironic, given that the original >> use of properties was for syntax-valued keys like 'origin. Properties >> like 'origin are meant to be extracted from just-expanded programs, >> though, as opposed to used in a later compilation step. At the same >> time, those uses of properties are why we can't just make >> `syntax-property` reject property values that are syntax objects. > > What are the implications of this for the Types as Macros/Turnstile > stuff that Stephen and Alex have been working on, given that they > use heavily use syntax objects in syntax properties as part of > attaching type information to identifiers? Are they doing something > to work around this and/or using properties in a way that avoids > this issue? Or is it likely that they’ll run into this for programs > that span multiple modules? > > (Obviously, I have a personal motive for asking this question, since > I’ve been playing with Turnstile and the techniques it uses myself. > So far I’ve been able to write programs that span multiple modules > without much issue, modulo an issue I asked about on this mailing > list a month or two back, but I’d like to know if I’m relying on > some sort of undefined behavior.) > > -- > You received this message because you are subscribed to the Google Groups > "Racket Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to racket-users+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [racket-users] syntax-property lost across module boundary (WAS: format-id doesn't preserve preserved?-edness of syntax-property?)
Le jeudi 15 décembre 2016 00:48:13 UTC+1, Alexis King a écrit : > without much issue, modulo an issue I asked about on this mailing > list a month or two back Here's (what I suspect is) the thread Alexis mentions, for those who feel curious: https://groups.google.com/forum/#!topic/racket-users/TGax2h8dVxs (Identifier equality of identifiers stashed in preserved syntax properties). -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [racket-users] syntax-property lost across module boundary (WAS: format-id doesn't preserve preserved?-edness of syntax-property?)
On Wed, Dec 14, 2016 at 03:50:28PM -0800, Alexis King wrote: > What are the implications of this for the Types as Macros/Turnstile ... > > (Obviously, I have a personal motive for asking this question, since Based on the names of functions in my example, you'd never guess how I ran into this problem. (Hint: It was implementing dependent types in Racket using Types as Macros.) -- William J. Bowman -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. signature.asc Description: PGP signature
Re: [racket-users] syntax-property lost across module boundary (WAS: format-id doesn't preserve preserved?-edness of syntax-property?)
> On Dec 14, 2016, at 7:14 AM, Matthew Flatt wrote: > > To summarize, don't try to attach syntax objects as property values > like that. I know this advice sounds ironic, given that the original > use of properties was for syntax-valued keys like 'origin. Properties > like 'origin are meant to be extracted from just-expanded programs, > though, as opposed to used in a later compilation step. At the same > time, those uses of properties are why we can't just make > `syntax-property` reject property values that are syntax objects. What are the implications of this for the Types as Macros/Turnstile stuff that Stephen and Alex have been working on, given that they use heavily use syntax objects in syntax properties as part of attaching type information to identifiers? Are they doing something to work around this and/or using properties in a way that avoids this issue? Or is it likely that they’ll run into this for programs that span multiple modules? (Obviously, I have a personal motive for asking this question, since I’ve been playing with Turnstile and the techniques it uses myself. So far I’ve been able to write programs that span multiple modules without much issue, modulo an issue I asked about on this mailing list a month or two back, but I’d like to know if I’m relying on some sort of undefined behavior.) -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [racket-users] syntax-property lost across module boundary (WAS: format-id doesn't preserve preserved?-edness of syntax-property?)
At Wed, 14 Dec 2016 10:43:34 -0700, Matthew Flatt wrote: > But right now, as Sam well knows I'd like to clarify that this is because Sam has been very helpful in making things better --- not that he's been asking for new things. -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [racket-users] syntax-property lost across module boundary (WAS: format-id doesn't preserve preserved?-edness of syntax-property?)
Yes, it's obviously easier to implement this new feature with the new expander. But right now, as Sam well knows, I'm open to patches only as long as they make things no slower and no bigger. :) The current implementation is reliably within a factor of 2 of the old one, but we're still working to close the gap. At Wed, 14 Dec 2016 11:09:12 -0500, Matthias Felleisen wrote: > > > On Dec 14, 2016, at 10:23 AM, Sam Tobin-Hochstadt > wrote: > > > > On Wed, Dec 14, 2016 at 10:14 AM, Matthew Flatt wrote: > >> More generally, changes to the scope of a > >> syntax object are not propagated to property values that happen to be > >> themselves syntax objects. When you put a syntax object into a > >> property, then you get whatever that syntax object meant at the time it > >> was attached as a property. > > > > This is a limitation of syntax properties that I've long had to work > > around in Typed Racket as well. Are there fundamental reasons why we > > couldn't have a kind of syntax property that scopes propaged to? Would > > this be something that would be easier to implement in the new > > expander? > > > > Sam > > > I can’t believe I fell for this again. Ryan has explained this more than once > to me . . . and yet. In this spirit, I support Sam’s question :-) > > -- > You received this message because you are subscribed to the Google Groups > "Racket Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to racket-users+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [racket-users] syntax-property lost across module boundary (WAS: format-id doesn't preserve preserved?-edness of syntax-property?)
> On Dec 14, 2016, at 10:23 AM, Sam Tobin-Hochstadt > wrote: > > On Wed, Dec 14, 2016 at 10:14 AM, Matthew Flatt wrote: >> More generally, changes to the scope of a >> syntax object are not propagated to property values that happen to be >> themselves syntax objects. When you put a syntax object into a >> property, then you get whatever that syntax object meant at the time it >> was attached as a property. > > This is a limitation of syntax properties that I've long had to work > around in Typed Racket as well. Are there fundamental reasons why we > couldn't have a kind of syntax property that scopes propaged to? Would > this be something that would be easier to implement in the new > expander? > > Sam I can’t believe I fell for this again. Ryan has explained this more than once to me . . . and yet. In this spirit, I support Sam’s question :-) -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [racket-users] syntax-property lost across module boundary (WAS: format-id doesn't preserve preserved?-edness of syntax-property?)
On Wed, Dec 14, 2016 at 10:14 AM, Matthew Flatt wrote: > More generally, changes to the scope of a > syntax object are not propagated to property values that happen to be > themselves syntax objects. When you put a syntax object into a > property, then you get whatever that syntax object meant at the time it > was attached as a property. This is a limitation of syntax properties that I've long had to work around in Typed Racket as well. Are there fundamental reasons why we couldn't have a kind of syntax property that scopes propaged to? Would this be something that would be easier to implement in the new expander? Sam -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [racket-users] syntax-property lost across module boundary (WAS: format-id doesn't preserve preserved?-edness of syntax-property?)
I think there are two problems in this example. The first problem is that preserved syntax properties were not actually preserved in bytecode for a syntax object that has no source location. (I'm assuming that Alexis's tests were run in DrRacket in it's default mode of compiling bytecode for `require`d files.) I'll push a repair for that bug. The second problem is that syntax objects as property values don't work the way that you want. That limitation is why, even with the property-presevation bug fixed, the `free-identifier=?` test fails. Unfortunately, that's a limitation of syntax objects and properties that's difficult to resolve. To explain a little more, consider the program https://gist.github.com/mflatt/d569d8f3a89e6815cf18df26adc11053 For now, look at just the `property` submodule. The `quote-syntax/add-example-property` macro is like `quote-syntax`, except that it adds an 'example property that holds the current time at compile time. An expression like (syntax-property (quote-syntax/add-example-property id) 'example) accesses that compile-time value at run time. If you compile the program to bytecode, the value shown by running the module later still reports the time that the module was compiled. Now consider the enclosing module, which uses the `property` submodule and accesses the time for both a freshly quoted term and the value of `id` as exported by the submodule: (require 'property) (syntax-property (quote-syntax/add-example-property new-id) 'example) (syntax-property id 'example) I think you won't be too surprised that the freshly quoted identifier has a newer time than the one reported for `id`. Even though both values were obtained from the same `compile-time` variable, each module compilation (submodule and enclosing module) gets a fresh instance of `compile-time`. In "format-id1.rkt", instead of `compile-time` bound to a timestamp, you have a quoted syntax object that you can image being implicitly lifted as (define-for-syntax compile-time #'id^) In much the same way that `(current-inexact-milliseconds)` in (define-for-syntax compile-time (current-inexact-milliseconds)) produces a different result for each subsequent compile-time use of the modules, `#'id^` produces a different syntax object depending on how its enclosing module is instantiated. Imagine taking the compiled bytecode, saving it in two places, and loading it as a module from each place; the `#'id` evaluations for the copies will produce identifiers that claim to be from different modules relative to the filesystem. So, here's the limitation: the implicit lifting that happens for a quoted syntax object, which enables a kind of re-evaluation of that syntax object in each module instance, doesn't happen for syntax-property values. More generally, changes to the scope of a syntax object are not propagated to property values that happen to be themselves syntax objects. When you put a syntax object into a property, then you get whatever that syntax object meant at the time it was attached as a property. That might still sound ok for your case, because you compile and use "format-id1.rkt" in-place. Still, the information attached to `#'id^` changes. While compiling the module, the attached binding information says "the current module being compiled". When compiling "format-id2.rkt", the attached binding information for that `#'id^` says "format-id1.rkt". To summarize, don't try to attach syntax objects as property values like that. I know this advice sounds ironic, given that the original use of properties was for syntax-valued keys like 'origin. Properties like 'origin are meant to be extracted from just-expanded programs, though, as opposed to used in a later compilation step. At the same time, those uses of properties are why we can't just make `syntax-property` reject property values that are syntax objects. At Tue, 13 Dec 2016 14:16:49 -0800, Alexis King wrote: > > On Dec 13, 2016, at 11:23 AM, 'William J. Bowman' via Racket Users > > wrote: > > > > Notice that #'x is not the same identifier as x, and thus does not > > have the same syntax-properties. > > I’m not convinced this is correct. > > If you insert println statements in the body of define^, inside the > with-syntax block, you get the following: > > (println (syntax-property #'x 'definition)) > (println (syntax-property-preserved? #'x 'definition)) > > ; => # > ; => #t > > The syntax properties are definitely maintained in templates used > by `syntax`. > > > Why does this issue only comes up across a file boundary? I suspect > > this has to do with the behavior of make-rename-transformer. Since > > #'x was not getting 'not-free-identifier=? correctly, > > make-rename-transformer was optimizing this away, although I thought > > I was preventing that. > > This is more interesting, and it was my first instinct, too. For > that reason, I replaced uses of make-rename-transformer with uses > of make-variable-like-transfor
Re: [racket-users] syntax-property lost across module boundary (WAS: format-id doesn't preserve preserved?-edness of syntax-property?)
> On Dec 13, 2016, at 11:23 AM, 'William J. Bowman' via Racket Users > wrote: > > Notice that #'x is not the same identifier as x, and thus does not > have the same syntax-properties. I’m not convinced this is correct. If you insert println statements in the body of define^, inside the with-syntax block, you get the following: (println (syntax-property #'x 'definition)) (println (syntax-property-preserved? #'x 'definition)) ; => # ; => #t The syntax properties are definitely maintained in templates used by `syntax`. > Why does this issue only comes up across a file boundary? I suspect > this has to do with the behavior of make-rename-transformer. Since > #'x was not getting 'not-free-identifier=? correctly, > make-rename-transformer was optimizing this away, although I thought > I was preventing that. This is more interesting, and it was my first instinct, too. For that reason, I replaced uses of make-rename-transformer with uses of make-variable-like-transformer from syntax/transformer, but the behavior was the same, so I don’t think this has to do with make-rename-transformer specifically. However, something more interesting happens. I changed the macro that define^ defines to insert some debugging calls on macro invocation: (define-syntax (id stx) (println (syntax-property #'x 'definition)) (println (syntax-property-preserved? #'x 'definition)) ((set!-transformer-procedure (make-variable-like-transformer #'x)) stx)) The output of this is much more interesting. When running the source module, format-id1.rkt, the result is as one would expect: ; => # ; => #t However, when using the defined identifier from format-id2.rkt, the result is different: ; => #f ; => #f This is extremely interesting, since the first example demonstrates that the syntax property is most definitely marked as preserved, yet it gets lost by the time format-id2 tries to get at it, despite the fact that it should be the same exact syntax object. My guess as to what’s going on here is that the issue is not with make-rename-transformer but with quote-syntax. The above program expands into a macro definition that has (quote-syntax x) in it, and the x has a preserved syntax property on it. However, that property is not actually preserved, despite being marked as preserved. This is different from other uses of preserved syntax properties that attach their syntax properties to expressions in the resulting program, rather than quoted pieces of syntax. This feels very confusing, and it certainly feels like a bug, but maybe Matthew has an explanation for this behavior that I don’t currently understand? Alexis -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.