Re: Precedence for reader extensions
On Tue, Feb 19, 2013 at 12:33 AM, Mark H Weaver m...@netris.org wrote: Mikael Djurfeldt mik...@djurfeldt.com writes: I propose to simplify this to only two levels: 1. %read-hash-procedures 2. predefined syntax It turns out that the change I propose above was already implemented in read.c. The effect just wasn't visible due to a bug in flush_ws which caused all #! to be erroneously removed if they exist as the outermost expression. In the attached diff, I've fixed the flush_ws bug and cleaned up some garbage code in scm_read_sharp which was unreachable. Can I push this into the repository? I don't think this would be sufficient. The problem is that tokens of the form #!symboldelimiter have become standardized. To name a few examples, both R6RS and R7RS define the reader directives #!fold-case and #!no-fold-case, R6RS has #!r6rs, and SRFI-105 has #!curly-infix. Guile also has #! ... !# block comments to help with the handling of executable scripts. In what sense is it not sufficient? In any case: The present diff doesn't remove any functionality or make performance worse. It only removes some inconsistent behavior. At the same time it allows support for mit-scheme #!optional and #!rest. Best regards, Mikael D. reader-fix.diff Description: Binary data
Re: [Guile-commits] GNU Guile branch, wip-rtl-cps, updated. v2.1.0-180-g0d0808a
Hello, Yes, I completely agree with this. I didn't do that immediately because I'm trying to get the infrastructure for the general case working. I plan to implement un-boxing in CPS. The real reason not to do it yet is that the tree-il-CPS compiler can't compile any examples that would actually need boxes. (But it will be able to soon!) Noah On Tue, Feb 19, 2013 at 12:53 AM, Mark H Weaver m...@netris.org wrote: Hi Noah, Noah Lavine noah.b.lav...@gmail.com writes: commit 0d0808ae3f7390ffb250b9deb6706ad4158cce0e Author: Noah Lavine noah.b.lav...@gmail.com Date: Mon Feb 18 14:10:58 2013 -0500 Make Lambda Arguments Mutable * module/language/cps.scm: let variable objects come with an initialization value. * module/language/tree-il/compile-cps.scm: put all lambda arguments in variable boxes, so they are mutable. Lambda arguments (and all other lexical variables) should only be put into boxes if they are 'set!' somewhere within their lexical scope. This can always be determined at compile time. It is crucial that we minimize the number of mutable variables, since they inhibit most optimizations. The required analysis is already implemented in tree-il/analyze.scm. Regards, Mark
Re: [Guile-commits] GNU Guile branch, wip-rtl-cps, updated. v2.1.0-180-g0d0808a
Oh, and thanks a lot for reviewing the CPS stuff! I really appreciate it, and I think it will make the end result a lot better than whatever I could do on my own. Noah On Tue, Feb 19, 2013 at 9:28 AM, Noah Lavine noah.b.lav...@gmail.comwrote: Hello, Yes, I completely agree with this. I didn't do that immediately because I'm trying to get the infrastructure for the general case working. I plan to implement un-boxing in CPS. The real reason not to do it yet is that the tree-il-CPS compiler can't compile any examples that would actually need boxes. (But it will be able to soon!) Noah On Tue, Feb 19, 2013 at 12:53 AM, Mark H Weaver m...@netris.org wrote: Hi Noah, Noah Lavine noah.b.lav...@gmail.com writes: commit 0d0808ae3f7390ffb250b9deb6706ad4158cce0e Author: Noah Lavine noah.b.lav...@gmail.com Date: Mon Feb 18 14:10:58 2013 -0500 Make Lambda Arguments Mutable * module/language/cps.scm: let variable objects come with an initialization value. * module/language/tree-il/compile-cps.scm: put all lambda arguments in variable boxes, so they are mutable. Lambda arguments (and all other lexical variables) should only be put into boxes if they are 'set!' somewhere within their lexical scope. This can always be determined at compile time. It is crucial that we minimize the number of mutable variables, since they inhibit most optimizations. The required analysis is already implemented in tree-il/analyze.scm. Regards, Mark
Re: Precedence for reader extensions
Mikael Djurfeldt mik...@djurfeldt.com writes: On Tue, Feb 19, 2013 at 12:33 AM, Mark H Weaver m...@netris.org wrote: Mikael Djurfeldt mik...@djurfeldt.com writes: I propose to simplify this to only two levels: 1. %read-hash-procedures 2. predefined syntax It turns out that the change I propose above was already implemented in read.c. The effect just wasn't visible due to a bug in flush_ws which caused all #! to be erroneously removed if they exist as the outermost expression. I'm not sure that I consider this a bug. All of the tokens that flush_ws removes can appear anywhere that whitespace is allowed, and are considered whitespace to the caller (although reader directives may modify the per-port reader options as a side-effect). #;expr (sexp-comments) #!fold-case (reader directives) #! ... !#(shebang block comments) #| ... |#(r6rs block comments) In the attached diff, I've fixed the flush_ws bug and cleaned up some garbage code in scm_read_sharp which was unreachable. Can I push this into the repository? I'm uncomfortable with globally overriding standard read syntax. In a large scheme system such as Guile, there are many modules that use 'read' and expect it to act in accordance with standard lexical conventions. Therefore, I'd prefer to limit 'read-hash-extend' to adding new syntax that would otherwise have been considered an error. If you're going to override standard read syntax, then I think it should only be done on a per-port basis. Therefore, I'd prefer a precedence closer to this: 1. (possibly) per-port variant of %read-hash-procedures 2. predefined syntax 3. %read-hash-procedures I don't think this would be sufficient. The problem is that tokens of the form #!symboldelimiter have become standardized. To name a few examples, both R6RS and R7RS define the reader directives #!fold-case and #!no-fold-case, R6RS has #!r6rs, and SRFI-105 has #!curly-infix. Guile also has #! ... !# block comments to help with the handling of executable scripts. In what sense is it not sufficient? In any case: The present diff doesn't remove any functionality or make performance worse. It only removes some inconsistent behavior. At the same time it allows support for mit-scheme #!optional and #!rest. The problem with this approach is that it does not compose. You want to add #!optional and #!rest. R6RS added #!r6rs, #!fold-case, and #!no-fold-case. SRFI-105 added #!curly-infix. But there can be only one read-hash-procedure for #!, and it's global to the entire system. That's why I suggested a way to add new tokens of the form #!symboldelimiter. That way, you could add handlers for #!rest and #!optional without interfering with the other #!symboldelimiter tokens. What do you think? Regards, Mark
Re: CPS Update
Hello, On Sat, Feb 16, 2013 at 4:18 PM, Mark H Weaver m...@netris.org wrote: Hi Noah, On Sat, Feb 16, 2013 at 2:14 PM, Mark H Weaver m...@netris.org wrote: [...] Noah Lavine noah.b.lav...@gmail.com writes: You mean if a function modifies another function that called it. There are many other cases. Think multiple threads, coroutines, logic programming systems, etc. That's why I wrote stack(s). Actually, I should have written (partial) continuation(s). There are any number of ways that an activation record for some procedure you modify could still be alive somewhere in the heap. The issue can arise even with simple lazy data structures. I don't think it's something we should punt on. IMO anyway. What do you think? Yes, you're right. I hadn't thought about those cases. This is a tricky question. But before we continue, are you sure that the right semantics is to modify all of the continuations? In particular, let's say you have a function like this: (define (func x) (+ x 2 (my-special-function x))) And my-special-function captures its continuation, k. Later on, you modify func to be this: (define (func x) (+ x 2)) Now what is the continuation k supposed to do? That continuation doesn't exist in the latest version of func. I think in this case you have to treat it like a closure that is still holding on to the continuation that it was passed (conceptually, at least) when it was called. So it would return to the old version of func. On the other hand, take the same example, but this time redefine + instead of func. Now, does the continuation k call the new definition of +, or the old one? These really are questions for me. I don't know what the correct behavior here is, but I think that if we can answer both of these questions, then we know more or less what the correct thing to do is. Noah
Re: [Guile-commits] GNU Guile branch, wip-rtl-cps, updated. v2.1.0-180-g0d0808a
Hi Noah, Noah Lavine noah.b.lav...@gmail.com writes: Yes, I completely agree with this. I didn't do that immediately because I'm trying to get the infrastructure for the general case working. I plan to implement un-boxing in CPS. You still seem to be proceeding from the assumption that the conversion to CPS will happen early, and that all optimizations will happen in CPS. I continue to think that this is a bad idea, because of the order of evaluation issue. I realize that you intend to extend CPS with some way to express unspecified evaluation order, but I'm not sure that is a good idea. The fact that CPS fully specifies evaluation order is not merely an undesirable flaw to be remedied. It is fundamental to the nature of CPS form, and an important part of what makes CPS desireable as an IR. I fear that in trying to get the best of both worlds, you will instead end up with the worst of both worlds. I suspect that the way to get the best of both worlds is to do several optimizations *before* conversion to CPS. We already have an increasingly sophisticated set of optimization passes implemented in tree-il. Those early passes already analyze whether or not lexicals need to be mutable or not, and make several optimizations that depend on having this information. Do you intend to rewrite all of those passes for CPS? Regards, Mark
Re: CPS Update
Noah Lavine noah.b.lav...@gmail.com writes: But before we continue, are you sure that the right semantics is to modify all of the continuations? In particular, let's say you have a function like this: (define (func x) (+ x 2 (my-special-function x))) And my-special-function captures its continuation, k. Later on, you modify func to be this: (define (func x) (+ x 2)) Now what is the continuation k supposed to do? That continuation doesn't exist in the latest version of func. I think in this case you have to treat it like a closure that is still holding on to the continuation that it was passed (conceptually, at least) when it was called. So it would return to the old version of func. Yes. If you redefine 'func', that only affects future calls to func, not existing calls. On the other hand, take the same example, but this time redefine + instead of func. Now, does the continuation k call the new definition of +, or the old one? In your example above, it's unspecified, because the operator and operands of a procedure call are evaluated in unspecified order. Therefore, an implementation is allowed to evaluate '+' either before or after it evaluates (my-special-function x). However, consider this slightly different example: (define (func x) (let ((r (my-special-function x))) (+ x 2 r))) Here, (my-special-function x) must be evaluated before evaluating '+'. Evaluating '+' means to fetch the value stored in the location denoted by '+'. Therefore, if '+' is rebound during the call to 'my-special-function', then the new binding for '+' must be used. This is a case where on-stack-replacement is needed to implement the correct semantics. To summarize, when you rebind a function 'foo', it is not the existing activation records for 'foo' that you need to worry about. Instead, you need to worry about existing activation records for all compiled procedures 'bar' that incorporated assumptions about 'foo'. Thanks for working on this, Mark
Re: Precedence for reader extensions
On Tue, Feb 19, 2013 at 4:41 PM, Mark H Weaver m...@netris.org wrote: Mikael Djurfeldt mik...@djurfeldt.com writes: On Tue, Feb 19, 2013 at 12:33 AM, Mark H Weaver m...@netris.org wrote: Mikael Djurfeldt mik...@djurfeldt.com writes: I propose to simplify this to only two levels: 1. %read-hash-procedures 2. predefined syntax It turns out that the change I propose above was already implemented in read.c. The effect just wasn't visible due to a bug in flush_ws which caused all #! to be erroneously removed if they exist as the outermost expression. I'm not sure that I consider this a bug. In this reply I've attached a file mit-reader-scm which installs a hash-read-procedure for #\!. What I wanted to say above is that scm_read_sharp (in HEAD) is implemented with the priorities I list above while flush_ws is implemented with other priorities. Here's a demo of the consequences of this bug: scheme@(guile-user) (load mit-reader.scm) scheme@(guile-user) (quote #!optional) ... !# hi) $1 = hi scheme@(guile-user) '#!optional $2 = #:optional [...] I'm uncomfortable with globally overriding standard read syntax. In a large scheme system such as Guile, there are many modules that use 'read' and expect it to act in accordance with standard lexical conventions. Well, in the mit-scheme compatibility module, my intention was to use dynamic-wind to modify #!-syntax while loading mit-scheme-specific files. Note that %read-hash-procedures is a fluid, so this will be absolutely local and won't leak out in any way to the rest of the system. The problem with this approach is that it does not compose. Let's now patch guile according to the diff I sent... there! scheme@(guile-user) (load mit-reader.scm) scheme@(guile-user) (quote #!optional) $1 = #:optional scheme@(guile-user) '#!optional $2 = #:optional scheme@(guile-user) (quote #!hi!# #!optional) $3 = #:optional My take on this is: * The %read-hash-procedures API is not pretty * The suggested change doesn't make things prettier * The suggested change *does* make things conceptually simpler and more flexible (= you can always override hash syntax if you want; compared to the current: you can override #| but not other hash syntax) * The suggested change fixes a bug * The suggested change does compose and different syntax can be confined to a module by using dynamic-wind Best regards, Mikael mit-reader.scm Description: Binary data
Re: [Guile-commits] GNU Guile branch, wip-rtl-cps, updated. v2.1.0-180-g0d0808a
Hello, On Tue, Feb 19, 2013 at 11:03 AM, Mark H Weaver m...@netris.org wrote: Hi Noah, Noah Lavine noah.b.lav...@gmail.com writes: Yes, I completely agree with this. I didn't do that immediately because I'm trying to get the infrastructure for the general case working. I plan to implement un-boxing in CPS. You still seem to be proceeding from the assumption that the conversion to CPS will happen early, and that all optimizations will happen in CPS. I continue to think that this is a bad idea, because of the order of evaluation issue. I realize that you intend to extend CPS with some way to express unspecified evaluation order, but I'm not sure that is a good idea. The fact that CPS fully specifies evaluation order is not merely an undesirable flaw to be remedied. It is fundamental to the nature of CPS form, and an important part of what makes CPS desireable as an IR. I fear that in trying to get the best of both worlds, you will instead end up with the worst of both worlds. I suspect that the way to get the best of both worlds is to do several optimizations *before* conversion to CPS. We already have an increasingly sophisticated set of optimization passes implemented in tree-il. Those early passes already analyze whether or not lexicals need to be mutable or not, and make several optimizations that depend on having this information. Do you intend to rewrite all of those passes for CPS? That's a fair point. I do think that some of those optimizations would work better in CPS, but even if that's true, I don't want to port them all at once. I agree that the goal should be to have a compiler in which some optimizations are done in Tree-IL and some in CPS, and once we have that, we can play with which optimizations happen where. So for now, yes, you are completely correct. However, you might like my other reason better: the Tree-IL-CPS compiler can't compile any cases that really need mutable variable slots, so I had to test mutable variables with examples that really don't need them. Once the Tree-IL-CPS compiler can handle interesting cases, then we can start optimizing the uninteresting ones away. Best, Noah
Re: Precedence for reader extensions
On Tue, Feb 19, 2013 at 5:42 PM, Mikael Djurfeldt mik...@djurfeldt.com wrote: * The suggested change does compose What I meant here is that it does compose with the built-in syntax. Of course, the %read-hash-procedures API by itself doesn't automatically compose if multiple user-defined modules use it to introduce new syntax. (If these modules take care to preserve previously installed procedures, it can compose.) The API you suggest would compose much easier, but to me it feels like just another specialized solution. What we would really need is something like Ludovic's guile-reader. But I won't be stubborn regarding this. If someone else wants to implement another way of supporting #!optional and #!rest that is fine by me. I regard my diff simply as a bug fix and cleanup (removing unreachable code).
Re: CPS Update
Hi, 1. Does any other system recompile stuff that set! a + operation 2. Isn't the least we should do to warn when a user set! + as I did in the last email 3. Wouldn't it be a good idea to allow a user to specify which basic vm op's would be translated to a macine instruction or not. So say that I would like to count the number of scheme + guile uses at startup, then I could specify in a configuration that + should not be comiled to a vm-op and then recompile guile and then use set! On Sat, Feb 16, 2013 at 5:29 PM, Noah Lavine noah.b.lav...@gmail.com wrote: Hello, On Sat, Feb 16, 2013 at 2:39 AM, Stefan Israelsson Tampe stefan.ita...@gmail.com wrote: Isn't the primitiveness decided by tree-il? you wil get a (primcall ...) tree il element and then should know how to emit a corresponding primitive instruction(s). Oh, you're right. I was thinking about that because I don't run the Tree-IL optimizers when I test it, so I don't get any Tree-IL primitives. Eventually maybe the primitive generation should happen in CPS, but I don't think it has to happen all at once. Anyway I think that your conclusion is right, it would not make sense to support overwriting these primitives dynamically, but scheme@(guile-user) (set! + (lambda x (apply * x))) scheme@(guile-user) (+ 2 3) $1 = 6 scheme@(guile-user) (define (f x y) (+ x y)) scheme@(guile-user) ,x f Disassembly of #procedure f (x y): 0(assert-nargs-ee/locals 2) ;; 2 args, 0 locals 2(local-ref 0) ;; `x' 4(local-ref 1) ;; `y' 6(add) 7(return) So things are not consistent! Good example! This looks like a bug to me. I think I read that loading GOOPS turns things like '+ into generic operations - that might be a case where this matters a lot. I have thought a bit about how to fix this. The module system already allows us to be notified whenever a variable changes, so it would be easy to watch all of the variables in (guile) and recompile procedures when they change. I might take a look at this soon. BTW If you like I could work on getting define* and lambda* to work, I have that code in my branch and should be able to make it work on your branch as well. WDYT? Thanks, but I haven't even gotten far enough to think about this. I'm still working on getting all of the basic features going. After that I would definitely like define* and lambda*. Noah /Stefan On Sat, Feb 16, 2013 at 1:53 AM, Noah Lavine noah.b.lav...@gmail.com wrote: Hello, The wip-rtl-cps branch has been rebased again (on top of wip-rtl). It now includes support for toplevel references and sets, thanks mostly to Andy's work on supporting them in RTL. (Although you shouldn't kick that part of it too hard just yet; I think I know where it will break.) Other highlights include using the top-level variable support to evaluate (+ 3 4) correctly. Overall, I think it's coming along well. The parts of Tree-IL that are left to implement are local mutable variables (should be easy after toplevel ones), module references and sets (very similar to top-level ones), closures, and the dynamic environment stuff. Local mutable variables are my next project, and then I think closures. One question I've had is, can we assume that the variables in the (guile) module are immutable? I think the current compiler does this. Otherwise there's no correct way to inline primitive procedures like + unless we have a way to invalidate our compiled code and recompile it (which I would like to have anyway, but that's a different story). Best, Noah
Re: Precedence for reader extensions
On Tue, Feb 19, 2013 at 5:42 PM, Mikael Djurfeldt mik...@djurfeldt.com wrote:
* The suggested change *does* make things conceptually simpler and
more flexible (= you can always override hash syntax if you want;
compared to the current: you can override #| but not other hash
syntax)
Just to try to be clear:
What I write above is not strictly true. The current Guile *already*
allows you to override standard syntax, even without my changes. What
my changes do is to cleanup the old mechanism so that it doesn't fail
when whitespace is involved.
An example of how it currently fails is that you *can* override when
you spell quote using single quote ('OBJECT) since no whitespace is
involved while you cannot override when you spell quote using the
symbol quote ((quote OBJECT)) since there's whitespace to be
swallowed before the OBJECT.
I do respect the attitude that the user shouldn't be able to override
standard syntax, even though I don't think it matters much given the
state of the current mess. But I think you agree that we either need
to apply my fix (making the current overriding mechanism useful) or
fix scm_read_sharp so that it conforms with the behavior of flush_ws.
