Re: [racket-dev] Trouble with state, submodules, and module-begin
On Mon, Jul 2, 2012 at 9:07 AM, Matthew Flatt mfl...@cs.utah.edu wrote: Thoughts so far: I think you need a new communication channel to get information from the expansion of an enclosing module to the expansion of its submodule. Expansion-time state is the right kind of channel, but I think it's important to start every submodule's expansion in a fresh store, at least usually. Otherwise, many syntactic extensions won't work correctly in a submodule. To give the programmer more control, we could add a `local-expand-submodule' function that is like `local-expand', but (1) it works only for `module' and `module*' forms in a 'module context, and (2) it accepts module paths to attach from the current expansion context to the submodule's expansion context. Using this addition, when expanding a `(module* name #f )' submodule, Typed Racket could attach a compile-time module that houses the in a typed context flag --- the same one that Typed Racket's `#%module-begin' sets. Does it sound like that would work? I believe that this would work, and I'll use it if it's provided. However, I don't (yet) think it's the right solution. In particular, I feel like this moves away from the really great feature of submodules, which is that they behave basically exactly like regular modules. From what I can tell, the only place where this uniformity breaks down currently is with `(module* name #f )', where the outer module is `require`d *before* the inner module starts expanding. In other words, if we have: (module M L (module* N #f ...)) currently `M` is required *before* `N` begins expanding, which is unlike any other module relationship that can be expressed in Racket (I think). You said in your earlier mail that you don't think it makes sense to change this ordering. Can you expand on the reasons why? I worry that `local-expand-submodule' might be used to break a syntactic form by attaching a module that isn't intended to be attached to multiple expansion stores. I think a macro that abuses `local-expand-submodule' could only harm itself or some other module that trusts the macro, but it's difficult to be sure. At Mon, 25 Jun 2012 16:47:36 -0600, Matthew Flatt wrote: At Mon, 25 Jun 2012 17:50:27 -0400, Sam Tobin-Hochstadt wrote: The problem (I think) is that the implicit `require` of `(submod ..)` happens *before* the expansion of `#%module-begin` inside the submodule. That's the same for a top-level module M whose initial language is some other module L, right? The require of `L' happens before the `#%module-begin' expansion in `M'... and it can't be any other way, because `#%module-begin' comes from `L'. The key bit of code is the residual snippet left in the outer module: (begin-for-syntax (when (unbox is-typed?) (set-box! type-env 1))) Currently, in TR, the code in the begin-for-syntax is unconditional, and therefore it gets re-run in the store used for expanding the inner submodule. However, if I add the `when`, then the `set-box!` doesn't happen, and the expansion of `m` fails. I'd like to be able to add this conditional, so I'd like to change the order of effects slightly here. I see what you mean, but I don't think it makes sense to change the order of things in the way that you're suggesting. I don't have any immediate ideas, but I'll think about it more. _ Racket Developers list: http://lists.racket-lang.org/dev -- sam th sa...@ccs.neu.edu _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Trouble with state, submodules, and module-begin
At Tue, 3 Jul 2012 11:05:52 -0400, Sam Tobin-Hochstadt wrote: However, I don't (yet) think it's the right solution. In particular, I feel like this moves away from the really great feature of submodules, which is that they behave basically exactly like regular modules. From what I can tell, the only place where this uniformity breaks down currently is with `(module* name #f )', where the outer module is `require`d *before* the inner module starts expanding. In other words, if we have: (module M L (module* N #f ...)) currently `M` is required *before* `N` begins expanding, which is unlike any other module relationship that can be expressed in Racket (I think). How does it differ from (module M L ) (module N M ) ? _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Trouble with state, submodules, and module-begin
On Tue, Jul 3, 2012 at 11:08 AM, Matthew Flatt mfl...@cs.utah.edu wrote: At Tue, 3 Jul 2012 11:05:52 -0400, Sam Tobin-Hochstadt wrote: However, I don't (yet) think it's the right solution. In particular, I feel like this moves away from the really great feature of submodules, which is that they behave basically exactly like regular modules. From what I can tell, the only place where this uniformity breaks down currently is with `(module* name #f )', where the outer module is `require`d *before* the inner module starts expanding. In other words, if we have: (module M L (module* N #f ...)) currently `M` is required *before* `N` begins expanding, which is unlike any other module relationship that can be expressed in Racket (I think). How does it differ from (module M L ) (module N M ) You're totally right. Good thing I added that (I think) qualifier. :) However, I think of the submodule case as more like: (module M L ) (module N L (require M) ) except with the ability to see more of `M` than `require` normally exposes. Thinking further about your example, I think that it would also break the conditional environment updates that I'm thinking of. I think that means that I need to go back to the drawing board and think about the requirements here. -- sam th sa...@ccs.neu.edu _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Trouble with state, submodules, and module-begin
On Mon, Jul 2, 2012 at 9:22 AM, Robby Findler ro...@eecs.northwestern.edu wrote: Is it possible there is another channel that TR could use to communicate these types? That is, could it not expand (: f Integer) (define f 5) into something that bound 'f' to a macro that knows its type? I guess it already does that, so something doesn't work about that here, but I'm not seeing what it is. Typed Racket doesn't do this -- `f` is bound to the same thing it would be in Racket, and a separate side table at expansion time maintains the type information, following the recipe Matthew originally described in the You Want it When? paper. I don't think the `special binding` version would work, for a number of reasons (full local-expansion, handling the base environment, expansion-time bindings). -- sam th sa...@ccs.neu.edu _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Trouble with state, submodules, and module-begin
At Tue, 3 Jul 2012 11:14:10 -0400, Sam Tobin-Hochstadt wrote: On Tue, Jul 3, 2012 at 11:08 AM, Matthew Flatt mfl...@cs.utah.edu wrote: At Tue, 3 Jul 2012 11:05:52 -0400, Sam Tobin-Hochstadt wrote: However, I don't (yet) think it's the right solution. In particular, I feel like this moves away from the really great feature of submodules, which is that they behave basically exactly like regular modules. From what I can tell, the only place where this uniformity breaks down currently is with `(module* name #f )', where the outer module is `require`d *before* the inner module starts expanding. In other words, if we have: (module M L (module* N #f ...)) currently `M` is required *before* `N` begins expanding, which is unlike any other module relationship that can be expressed in Racket (I think). How does it differ from (module M L ) (module N M ) You're totally right. Good thing I added that (I think) qualifier. :) However, I think of the submodule case as more like: (module M L ) (module N L (require M) ) except with the ability to see more of `M` than `require` normally exposes. It's really more like `(module N M ...)'. Here's an example to illustrate the difference: #lang racket/base (define-syntax-rule (#%module-begin) ; shadows `racket/base' version (#%plain-module-begin (displayln Hi))) (module+ main) ; prints Hi Thinking further about your example, I think that it would also break the conditional environment updates that I'm thinking of. I think that means that I need to go back to the drawing board and think about the requirements here. Ok. _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Trouble with state, submodules, and module-begin
Thoughts so far: I think you need a new communication channel to get information from the expansion of an enclosing module to the expansion of its submodule. Expansion-time state is the right kind of channel, but I think it's important to start every submodule's expansion in a fresh store, at least usually. Otherwise, many syntactic extensions won't work correctly in a submodule. To give the programmer more control, we could add a `local-expand-submodule' function that is like `local-expand', but (1) it works only for `module' and `module*' forms in a 'module context, and (2) it accepts module paths to attach from the current expansion context to the submodule's expansion context. Using this addition, when expanding a `(module* name #f )' submodule, Typed Racket could attach a compile-time module that houses the in a typed context flag --- the same one that Typed Racket's `#%module-begin' sets. Does it sound like that would work? I worry that `local-expand-submodule' might be used to break a syntactic form by attaching a module that isn't intended to be attached to multiple expansion stores. I think a macro that abuses `local-expand-submodule' could only harm itself or some other module that trusts the macro, but it's difficult to be sure. At Mon, 25 Jun 2012 16:47:36 -0600, Matthew Flatt wrote: At Mon, 25 Jun 2012 17:50:27 -0400, Sam Tobin-Hochstadt wrote: The problem (I think) is that the implicit `require` of `(submod ..)` happens *before* the expansion of `#%module-begin` inside the submodule. That's the same for a top-level module M whose initial language is some other module L, right? The require of `L' happens before the `#%module-begin' expansion in `M'... and it can't be any other way, because `#%module-begin' comes from `L'. The key bit of code is the residual snippet left in the outer module: (begin-for-syntax (when (unbox is-typed?) (set-box! type-env 1))) Currently, in TR, the code in the begin-for-syntax is unconditional, and therefore it gets re-run in the store used for expanding the inner submodule. However, if I add the `when`, then the `set-box!` doesn't happen, and the expansion of `m` fails. I'd like to be able to add this conditional, so I'd like to change the order of effects slightly here. I see what you mean, but I don't think it makes sense to change the order of things in the way that you're suggesting. I don't have any immediate ideas, but I'll think about it more. _ Racket Developers list: http://lists.racket-lang.org/dev _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Trouble with state, submodules, and module-begin
Is it possible there is another channel that TR could use to communicate these types? That is, could it not expand (: f Integer) (define f 5) into something that bound 'f' to a macro that knows its type? I guess it already does that, so something doesn't work about that here, but I'm not seeing what it is. Robby On Mon, Jul 2, 2012 at 8:07 AM, Matthew Flatt mfl...@cs.utah.edu wrote: Thoughts so far: I think you need a new communication channel to get information from the expansion of an enclosing module to the expansion of its submodule. Expansion-time state is the right kind of channel, but I think it's important to start every submodule's expansion in a fresh store, at least usually. Otherwise, many syntactic extensions won't work correctly in a submodule. To give the programmer more control, we could add a `local-expand-submodule' function that is like `local-expand', but (1) it works only for `module' and `module*' forms in a 'module context, and (2) it accepts module paths to attach from the current expansion context to the submodule's expansion context. Using this addition, when expanding a `(module* name #f )' submodule, Typed Racket could attach a compile-time module that houses the in a typed context flag --- the same one that Typed Racket's `#%module-begin' sets. Does it sound like that would work? I worry that `local-expand-submodule' might be used to break a syntactic form by attaching a module that isn't intended to be attached to multiple expansion stores. I think a macro that abuses `local-expand-submodule' could only harm itself or some other module that trusts the macro, but it's difficult to be sure. At Mon, 25 Jun 2012 16:47:36 -0600, Matthew Flatt wrote: At Mon, 25 Jun 2012 17:50:27 -0400, Sam Tobin-Hochstadt wrote: The problem (I think) is that the implicit `require` of `(submod ..)` happens *before* the expansion of `#%module-begin` inside the submodule. That's the same for a top-level module M whose initial language is some other module L, right? The require of `L' happens before the `#%module-begin' expansion in `M'... and it can't be any other way, because `#%module-begin' comes from `L'. The key bit of code is the residual snippet left in the outer module: (begin-for-syntax (when (unbox is-typed?) (set-box! type-env 1))) Currently, in TR, the code in the begin-for-syntax is unconditional, and therefore it gets re-run in the store used for expanding the inner submodule. However, if I add the `when`, then the `set-box!` doesn't happen, and the expansion of `m` fails. I'd like to be able to add this conditional, so I'd like to change the order of effects slightly here. I see what you mean, but I don't think it makes sense to change the order of things in the way that you're suggesting. I don't have any immediate ideas, but I'll think about it more. _ Racket Developers list: http://lists.racket-lang.org/dev _ Racket Developers list: http://lists.racket-lang.org/dev _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Trouble with state, submodules, and module-begin
The expansion of submodules is intended to have a fresh store, just like the expansion of any module. The `#f' initial import is intended to be something like using `(submod ..)' as the initial language, except importing all internal bindings of the module instead of its exports. Can Typed Racket not handle enclosing bindings in the same way that it handles `require's in general? At Mon, 25 Jun 2012 17:05:21 -0400, Sam Tobin-Hochstadt wrote: It appears that the expansion of (module* m #f ...) occurs in a fresh store, but before running the inner module's `#%module-begin` binding. To see this, run the program at https://gist.github.com/2991214 , and note that in module-begin is printed only once, but the expansion of `m` fails, indicating a fresh store. This is troublesome for Typed Racket, which uses the store to record the types of identifiers. Is this the intended behavior? -- sam th sa...@ccs.neu.edu _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Trouble with state, submodules, and module-begin
On Mon, Jun 25, 2012 at 5:14 PM, Matthew Flatt mfl...@cs.utah.edu wrote: The expansion of submodules is intended to have a fresh store, just like the expansion of any module. The `#f' initial import is intended to be something like using `(submod ..)' as the initial language, except importing all internal bindings of the module instead of its exports. Can Typed Racket not handle enclosing bindings in the same way that it handles `require's in general? Currently, it can. However, I wanted to make a fairly small change to only update the type environment when needed, and it broke because of this. In the context of my example, the `(m)` invocation is expanded *before* the inner `#%module-begin` expansion happens, meaning that it hits the error case, even though it's expanded inside a module with the appropriate `#%module-begin` binding. With submodules as they currently work, the inner `#%module-begin` is expanded quite late, which I think breaks other possible expectations about #%module-begin as well. Moving it earlier will, I think, alleviate all of these issues. At Mon, 25 Jun 2012 17:05:21 -0400, Sam Tobin-Hochstadt wrote: It appears that the expansion of (module* m #f ...) occurs in a fresh store, but before running the inner module's `#%module-begin` binding. To see this, run the program at https://gist.github.com/2991214 , and note that in module-begin is printed only once, but the expansion of `m` fails, indicating a fresh store. This is troublesome for Typed Racket, which uses the store to record the types of identifiers. Is this the intended behavior? -- sam th sa...@ccs.neu.edu -- sam th sa...@ccs.neu.edu _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Trouble with state, submodules, and module-begin
I think I misunderstood your complaint. If you change the example to (module+ sub (#%module-begin (m))) then there's no error. So, I think you're running into the old only-form-in-module-just-might-expand-to-#%module-begin problem. The usual solution is form `m' to check whether it's being expanded in a 'module-begin context, in which case it should wrap itself in `begin', or something like that. At Mon, 25 Jun 2012 17:28:23 -0400, Sam Tobin-Hochstadt wrote: On Mon, Jun 25, 2012 at 5:14 PM, Matthew Flatt mfl...@cs.utah.edu wrote: The expansion of submodules is intended to have a fresh store, just like the expansion of any module. The `#f' initial import is intended to be something like using `(submod ..)' as the initial language, except importing all internal bindings of the module instead of its exports. Can Typed Racket not handle enclosing bindings in the same way that it handles `require's in general? Currently, it can. However, I wanted to make a fairly small change to only update the type environment when needed, and it broke because of this. In the context of my example, the `(m)` invocation is expanded *before* the inner `#%module-begin` expansion happens, meaning that it hits the error case, even though it's expanded inside a module with the appropriate `#%module-begin` binding. With submodules as they currently work, the inner `#%module-begin` is expanded quite late, which I think breaks other possible expectations about #%module-begin as well. Moving it earlier will, I think, alleviate all of these issues. At Mon, 25 Jun 2012 17:05:21 -0400, Sam Tobin-Hochstadt wrote: It appears that the expansion of (module* m #f ...) occurs in a fresh store, but before running the inner module's `#%module-begin` binding. To see this, run the program at https://gist.github.com/2991214 , and note that in module-begin is printed only once, but the expansion of `m` fails, indicating a fresh store. This is troublesome for Typed Racket, which uses the store to record the types of identifiers. Is this the intended behavior? -- sam th sa...@ccs.neu.edu -- sam th sa...@ccs.neu.edu _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Trouble with state, submodules, and module-begin
On Mon, Jun 25, 2012 at 5:38 PM, Matthew Flatt mfl...@cs.utah.edu wrote: I think I misunderstood your complaint. If you change the example to (module+ sub (#%module-begin (m))) then there's no error. So, I think you're running into the old only-form-in-module-just-might-expand-to-#%module-begin problem. Sorry, that was a confusing red herring on my part. I now think I understand what's going wrong. I've revised the gist to be a more complete representation of what's going on in TR. The problem (I think) is that the implicit `require` of `(submod ..)` happens *before* the expansion of `#%module-begin` inside the submodule. The key bit of code is the residual snippet left in the outer module: (begin-for-syntax (when (unbox is-typed?) (set-box! type-env 1))) Currently, in TR, the code in the begin-for-syntax is unconditional, and therefore it gets re-run in the store used for expanding the inner submodule. However, if I add the `when`, then the `set-box!` doesn't happen, and the expansion of `m` fails. I'd like to be able to add this conditional, so I'd like to change the order of effects slightly here. The usual solution is form `m' to check whether it's being expanded in a 'module-begin context, in which case it should wrap itself in `begin', or something like that. At Mon, 25 Jun 2012 17:28:23 -0400, Sam Tobin-Hochstadt wrote: On Mon, Jun 25, 2012 at 5:14 PM, Matthew Flatt mfl...@cs.utah.edu wrote: The expansion of submodules is intended to have a fresh store, just like the expansion of any module. The `#f' initial import is intended to be something like using `(submod ..)' as the initial language, except importing all internal bindings of the module instead of its exports. Can Typed Racket not handle enclosing bindings in the same way that it handles `require's in general? Currently, it can. However, I wanted to make a fairly small change to only update the type environment when needed, and it broke because of this. In the context of my example, the `(m)` invocation is expanded *before* the inner `#%module-begin` expansion happens, meaning that it hits the error case, even though it's expanded inside a module with the appropriate `#%module-begin` binding. With submodules as they currently work, the inner `#%module-begin` is expanded quite late, which I think breaks other possible expectations about #%module-begin as well. Moving it earlier will, I think, alleviate all of these issues. At Mon, 25 Jun 2012 17:05:21 -0400, Sam Tobin-Hochstadt wrote: It appears that the expansion of (module* m #f ...) occurs in a fresh store, but before running the inner module's `#%module-begin` binding. To see this, run the program at https://gist.github.com/2991214 , and note that in module-begin is printed only once, but the expansion of `m` fails, indicating a fresh store. This is troublesome for Typed Racket, which uses the store to record the types of identifiers. Is this the intended behavior? -- sam th sa...@ccs.neu.edu -- sam th sa...@ccs.neu.edu -- sam th sa...@ccs.neu.edu _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Trouble with state, submodules, and module-begin
At Mon, 25 Jun 2012 17:50:27 -0400, Sam Tobin-Hochstadt wrote: The problem (I think) is that the implicit `require` of `(submod ..)` happens *before* the expansion of `#%module-begin` inside the submodule. That's the same for a top-level module M whose initial language is some other module L, right? The require of `L' happens before the `#%module-begin' expansion in `M'... and it can't be any other way, because `#%module-begin' comes from `L'. The key bit of code is the residual snippet left in the outer module: (begin-for-syntax (when (unbox is-typed?) (set-box! type-env 1))) Currently, in TR, the code in the begin-for-syntax is unconditional, and therefore it gets re-run in the store used for expanding the inner submodule. However, if I add the `when`, then the `set-box!` doesn't happen, and the expansion of `m` fails. I'd like to be able to add this conditional, so I'd like to change the order of effects slightly here. I see what you mean, but I don't think it makes sense to change the order of things in the way that you're suggesting. I don't have any immediate ideas, but I'll think about it more. _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Trouble with state, submodules, and module-begin
Sam, can you focus on cases of submodules for which you can guarantee that #%module-begin is okay and reject others [with an explicit, informative error message]? -- Matthias On Jun 25, 2012, at 6:47 PM, Matthew Flatt wrote: At Mon, 25 Jun 2012 17:50:27 -0400, Sam Tobin-Hochstadt wrote: The problem (I think) is that the implicit `require` of `(submod ..)` happens *before* the expansion of `#%module-begin` inside the submodule. That's the same for a top-level module M whose initial language is some other module L, right? The require of `L' happens before the `#%module-begin' expansion in `M'... and it can't be any other way, because `#%module-begin' comes from `L'. The key bit of code is the residual snippet left in the outer module: (begin-for-syntax (when (unbox is-typed?) (set-box! type-env 1))) Currently, in TR, the code in the begin-for-syntax is unconditional, and therefore it gets re-run in the store used for expanding the inner submodule. However, if I add the `when`, then the `set-box!` doesn't happen, and the expansion of `m` fails. I'd like to be able to add this conditional, so I'd like to change the order of effects slightly here. I see what you mean, but I don't think it makes sense to change the order of things in the way that you're suggesting. I don't have any immediate ideas, but I'll think about it more. _ Racket Developers list: http://lists.racket-lang.org/dev _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Trouble with state, submodules, and module-begin
On Mon, Jun 25, 2012 at 6:53 PM, Matthias Felleisen matth...@ccs.neu.edu wrote: Sam, can you focus on cases of submodules for which you can guarantee that #%module-begin is okay and reject others [with an explicit, informative error message]? -- Matthias I'm not entirely sure what you mean here, but: (a) this discussion is just about enabling an optimization, which I thus won't for the immediate future. This is an important optimization, and one that's a step towards removing the overhead of using Typed Racket, but it's still an optimization and I can do without it. (b) the affected submodules include everything that uses `module+`, so rejecting those isn't really an option. Sam On Jun 25, 2012, at 6:47 PM, Matthew Flatt wrote: At Mon, 25 Jun 2012 17:50:27 -0400, Sam Tobin-Hochstadt wrote: The problem (I think) is that the implicit `require` of `(submod ..)` happens *before* the expansion of `#%module-begin` inside the submodule. That's the same for a top-level module M whose initial language is some other module L, right? The require of `L' happens before the `#%module-begin' expansion in `M'... and it can't be any other way, because `#%module-begin' comes from `L'. The key bit of code is the residual snippet left in the outer module: (begin-for-syntax (when (unbox is-typed?) (set-box! type-env 1))) Currently, in TR, the code in the begin-for-syntax is unconditional, and therefore it gets re-run in the store used for expanding the inner submodule. However, if I add the `when`, then the `set-box!` doesn't happen, and the expansion of `m` fails. I'd like to be able to add this conditional, so I'd like to change the order of effects slightly here. I see what you mean, but I don't think it makes sense to change the order of things in the way that you're suggesting. I don't have any immediate ideas, but I'll think about it more. _ Racket Developers list: http://lists.racket-lang.org/dev -- sam th sa...@ccs.neu.edu _ Racket Developers list: http://lists.racket-lang.org/dev
