Re: [racket-dev] syntax taints instead of syntax certificates
On Saturday, Matthew Flatt wrote: At Sat, 2 Jul 2011 12:34:49 -0400, Eli Barzilay wrote: I read the guide chapter -- it has a few typos: 1. The page is stx-certs.html. Intentional, though I'm not sure it's a good idea. There's a small benefit of keeping the URL the same and making old links to the certificate chapter point to the taint chapter; that small benefit seemed to me to outweigh the benefit of making the URL and section-reference tag say taint. (FWIW, I've considered changing the encoding used -- there's more and more sites that use a common markdown which has issues with parens in the URL. But OTOH there are links to these pages because of that, so possibly do that with some rewrite rules to forward the old links.) Also, I still find the relatively large number of as something that makes understanding it hard, and maybe it's because I'm not a native, but I find the taints/dye-pack confusing and unhelpful. A large number of technical terms? Large number of terms that are supposed to work as an analogy: taints, dye-packs, arm/disarm, protect. It bothers me too, and I tried to write the documentation with fewer terms before, but I think it turns out better with more terms. Ryan and I spent a lot of time refining the terminology, but I guess it could always be better. [...] I really liked that these things didn't work before. It's good to know that there was a hole that is now plugable, but it looks to me like the cost is getting many more holes by mistake. I think I understand how you define hole, but just to be clear, I think the switch to taints strictly decreases the number of macros that can be abused --- since any macro that can be abused after the taint change could always be abused via `syntax-local-expand'. IIUC, all macros could have been abused previously, no? If so, then obviously new situation has fewer abusable macros. But there *were* some checks before which are now not done by default, right? To put this differently, you mentioned that things are a little smaller/faster now -- maybe this is a result of the checks that are now not done? (Or mostly not done.) Yes, I realize that. It looks to me like if all transformer results are protected (except for results explicitly tagged for exclusion), then we're at least getting the same automatic protection we got before, and with the same security hole. If so -- is it a good idea to drop protections just because there is a known way of getting around them? It's not obvious to me either way. It may turn out that not arming a macro expansion helps keep the taint system out of your way when you don't need it. That benefit could outweigh the benefit of blocking casual attempts at abuse, and I'm inclined to explore the possibility for now. But what I'm suggesting keeps the option of writing an unprotected macro, so I don't see what new benefits there are for not doing the automatic protection thing. BTW, there's another thing that is lost -- the easy way of switching between (define-syntax foo (syntax-rules () [(_ x) (blah blah)])) and (define-syntax (foo stx) (syntax-case stx () [(_ x) #'(blah blah)])) A careful programmer will need to know about the lack of protection, so in that sense the new setup is something that you need to deal with explicitly much more than before. Also, isn't it possible to somehow protect that #'bar so that you can't get its `syntax-local-value' without a powerful enough inspector or something along this line? Do you mean `protect-out'? Maybe something like that -- if it's posible to make it so that the expander can expand (bar ...) but you can't access the binding directly? -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: http://barzilay.org/ Maze is Life! _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] syntax taints instead of syntax certificates
On 06/30/2011 05:01 AM, Carl Eastlund wrote: On Wed, Jun 29, 2011 at 10:55 PM, Matthew Flattmfl...@cs.utah.edu wrote: The problem with having the macro transformer add `syntax-protect' for (define-syntax bar (lambda (stx) ...)) is that some other transformer can say ((syntax-local-value #'bar) stx) to get the transformer's result without `syntax-protect' --- which was a gaping hole that Ryan noticed in the certificate system. We considered ways of automating `syntax-protect' for all macros, but the ways we found seemed to create more problems than they solved. Thank you, that greatly clarifies the issue for me. Of course I would like to see syntax-case and syntax-parse have options to automatically syntax-protect as well; that will cover a lot more common cases for me. But now I can see where the trade-offs are that resulted in this design. Something like this springs to mind: (define-syntax provide/protection (syntax-rules () ((_ name) (begin (define-syntax tmp (syntax-rules () ((_ . args) (name . args (provide (rename-out (tmp name))) If I understand correctly then (provide/protection mymacro) would automatically wrap mymacro and export it (by virtue of redirecting the transformation through syntax-rules). May I suggest the following introduction for the documentation? Nice macros you have here. Would be a pity if ... something happened to them. But we can provide... protection. Stephan _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] syntax taints instead of syntax certificates
On Thu, Jun 30, 2011 at 2:20 PM, Stephan Houben steph...@planet.nl wrote: Something like this springs to mind: (define-syntax provide/protection (syntax-rules () ((_ name) (begin (define-syntax tmp (syntax-rules () ((_ . args) (name . args (provide (rename-out (tmp name))) If I understand correctly then (provide/protection mymacro) would automatically wrap mymacro and export it (by virtue of redirecting the transformation through syntax-rules). Oh, excellent. That should be straightforward enough. Then we have provide/contract for values and provide/protect for macros, and all invariants are protected. :D May I suggest the following introduction for the documentation? Nice macros you have here. Would be a pity if ... something happened to them. But we can provide... protection. ... said Matthew menacingly. Stephan watched with a growing sense of dread as Matthew's enforcer Ryan picked up a carefully placed identifier and threw it into an unsafe context. --Carl _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] syntax taints instead of syntax certificates
I read the guide chapter -- it has a few typos: 1. The page is stx-certs.html. 2. There's still one mention of certificate. 3. Some modules are referred to by name where there is no such name. For example, the q module (I'm pretty sure that there were more). 4. Talks about defining `protect-go-syntax', but it should be `go-syntax-protect'. Also, I still find the relatively large number of as something that makes understanding it hard, and maybe it's because I'm not a native, but I find the taints/dye-pack confusing and unhelpful. On Wednesday, Matthew Flatt wrote: I think you want to use `syntax-protect' on any exported macro. This does sound like what I was worried about. Here's an example that shows that `enter!' is now more broken than it was: Welcome to Racket v5.1.1.8. ((eval (cadr (syntax-list (expand-syntax #'(enter! foo) 1 2 3) procedure do-enter!: expects 2 arguments, given 3: 1 2 3 (hash-set! (eval (datum-syntax (cadr (syntax-list (expand-syntax #'(enter! foo 'loaded)) (collection-file-path main.rkt racket) 'junk) (enter! (file /tmp/x)) mod-depends: expects argument of type struct:mod; given 'junk I really liked that these things didn't work before. It's good to know that there was a hole that is now plugable, but it looks to me like the cost is getting many more holes by mistake. On Wed, Jun 29, 2011 at 10:10 PM, Eli Barzilay e...@barzilay.org wrote: An attempt to clarify what's unclear to me: if I'm supposed to almost always arm the result of a transformer, then why not have the expander do that for me, and add some new thing so in the exceptional cases I could do (define-syntax (foo stx) (dont-arm ...blah...)) ? Sorry --- that point was buried too deeply in my message (first sub-bullet for the last bullet in the third set of bullets). The problem with having the macro transformer add `syntax-protect' for (define-syntax bar (lambda (stx) ...)) is that some other transformer can say ((syntax-local-value #'bar) stx) to get the transformer's result without `syntax-protect' --- which was a gaping hole that Ryan noticed in the certificate system. We considered ways of automating `syntax-protect' for all macros, but the ways we found seemed to create more problems than they solved. Yes, I realize that. It looks to me like if all transformer results are protected (except for results explicitly tagged for exclusion), then we're at least getting the same automatic protection we got before, and with the same security hole. If so -- is it a good idea to drop protections just because there is a known way of getting around them? (At least as far as an army goes, I know what the answer to that would be...) Also, isn't it possible to somehow protect that #'bar so that you can't get its `syntax-local-value' without a powerful enough inspector or something along this line? -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: http://barzilay.org/ Maze is Life! _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] syntax taints instead of syntax certificates
At Sat, 2 Jul 2011 12:34:49 -0400, Eli Barzilay wrote: I read the guide chapter -- it has a few typos: 1. The page is stx-certs.html. Intentional, though I'm not sure it's a good idea. There's a small benefit of keeping the URL the same and making old links to the certificate chapter point to the taint chapter; that small benefit seemed to me to outweigh the benefit of making the URL and section-reference tag say taint. 2. There's still one mention of certificate. 3. Some modules are referred to by name where there is no such name. For example, the q module (I'm pretty sure that there were more). 4. Talks about defining `protect-go-syntax', but it should be `go-syntax-protect'. Fixed. Also, I still find the relatively large number of as something that makes understanding it hard, and maybe it's because I'm not a native, but I find the taints/dye-pack confusing and unhelpful. A large number of technical terms? It bothers me too, and I tried to write the documentation with fewer terms before, but I think it turns out better with more terms. Ryan and I spent a lot of time refining the terminology, but I guess it could always be better. On Wednesday, Matthew Flatt wrote: I think you want to use `syntax-protect' on any exported macro. This does sound like what I was worried about. Here's an example that shows that `enter!' is now more broken than it was: Welcome to Racket v5.1.1.8. ((eval (cadr (syntax-list (expand-syntax #'(enter! foo) 1 2 3) procedure do-enter!: expects 2 arguments, given 3: 1 2 3 (hash-set! (eval (datum-syntax (cadr (syntax-list (expand-syntax #'(enter! foo 'loaded)) (collection-file-path main.rkt racket) 'junk) (enter! (file /tmp/x)) mod-depends: expects argument of type struct:mod; given 'junk I've added `syntax-protect' for `enter!'. I really liked that these things didn't work before. It's good to know that there was a hole that is now plugable, but it looks to me like the cost is getting many more holes by mistake. I think I understand how you define hole, but just to be clear, I think the switch to taints strictly decreases the number of macros that can be abused --- since any macro that can be abused after the taint change could always be abused via `syntax-local-expand'. On Wed, Jun 29, 2011 at 10:10 PM, Eli Barzilay e...@barzilay.org wrote: An attempt to clarify what's unclear to me: if I'm supposed to almost always arm the result of a transformer, then why not have the expander do that for me, and add some new thing so in the exceptional cases I could do (define-syntax (foo stx) (dont-arm ...blah...)) ? Sorry --- that point was buried too deeply in my message (first sub-bullet for the last bullet in the third set of bullets). The problem with having the macro transformer add `syntax-protect' for (define-syntax bar (lambda (stx) ...)) is that some other transformer can say ((syntax-local-value #'bar) stx) to get the transformer's result without `syntax-protect' --- which was a gaping hole that Ryan noticed in the certificate system. We considered ways of automating `syntax-protect' for all macros, but the ways we found seemed to create more problems than they solved. Yes, I realize that. It looks to me like if all transformer results are protected (except for results explicitly tagged for exclusion), then we're at least getting the same automatic protection we got before, and with the same security hole. If so -- is it a good idea to drop protections just because there is a known way of getting around them? It's not obvious to me either way. It may turn out that not arming a macro expansion helps keep the taint system out of your way when you don't need it. That benefit could outweigh the benefit of blocking casual attempts at abuse, and I'm inclined to explore the possibility for now. Also, isn't it possible to somehow protect that #'bar so that you can't get its `syntax-local-value' without a powerful enough inspector or something along this line? Do you mean `protect-out'? _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] syntax taints instead of syntax certificates
On Wed, Jun 29, 2011 at 9:19 PM, Matthew Flatt mfl...@cs.utah.edu wrote: I've pushed a change to Racket's macro system that throws out the syntax-certificate system and adds a syntax-taint system. Syntax taints, like syntax certificates before, are intended to protect macro expansions from abuse. Abuse means using `local-expand' to extract a piece of an expansion, then putting the piece in a new context or using `datum-syntax' to access unexported module bindings using the piece's lexical context. Meanwhile, program-processing tools like `errortrace' or languages like `typed/racket' are supposed to pull apart expansions and reorganize them; code inspectors remain the way to distinguish trusted tools and languages from potential abusers. Things you need to know: * When writing a macro (without `syntax-rules', `define-syntax-rule', or `syntax-id-rules'), apply `syntax-protect' to the macro's result syntax object to protect it from abuse. The `syntax-rules', etc., forms use `syntax-protect' automatically. Does this mean we should be updating all our old macros now, and adding syntax-protect as boilerplate to (nearly) all of our new macros? How often is a manual syntax-protect necessary? It's great to have a system we can reason about, but the nice thing about certificates was that they were invisible most of the time. It looks like you're suggesting that at least the use of syntax-protect is going to become ubiquitous in this new system. * If you write program-processing tools or languages that use `syntax-recertify', you'll need to change them to use `syntax-disarm' and possibly `syntax-rearm'. --Carl _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] syntax taints instead of syntax certificates
8 minutes ago, Carl Eastlund wrote: On Wed, Jun 29, 2011 at 9:19 PM, Matthew Flatt mfl...@cs.utah.edu wrote: I've pushed a change to Racket's macro system that throws out the syntax-certificate system and adds a syntax-taint system. Syntax taints, like syntax certificates before, are intended to protect macro expansions from abuse. Abuse means using `local-expand' to extract a piece of an expansion, then putting the piece in a new context or using `datum-syntax' to access unexported module bindings using the piece's lexical context. Meanwhile, program-processing tools like `errortrace' or languages like `typed/racket' are supposed to pull apart expansions and reorganize them; code inspectors remain the way to distinguish trusted tools and languages from potential abusers. Things you need to know: * When writing a macro (without `syntax-rules', `define-syntax-rule', or `syntax-id-rules'), apply `syntax-protect' to the macro's result syntax object to protect it from abuse. The `syntax-rules', etc., forms use `syntax-protect' automatically. Does this mean we should be updating all our old macros now, and adding syntax-protect as boilerplate to (nearly) all of our new macros? How often is a manual syntax-protect necessary? It's great to have a system we can reason about, but the nice thing about certificates was that they were invisible most of the time. It looks like you're suggesting that at least the use of syntax-protect is going to become ubiquitous in this new system. That's exactly what I'm worried about too. (I was going to try things out first, but the build crashed.) Actually, the certificate system was good enough at being invisible that explaining the new system in terms of it isn't effective (at least for me) since certificates were this complicated thing that you need to be aware of in some rare cases but 5 minutes after you're done you'd re-forget what you read. (Again, this is all for a subjective value of you, of course.) -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: http://barzilay.org/ Maze is Life! _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] syntax taints instead of syntax certificates
A few seconds ago, Eli Barzilay wrote: 8 minutes ago, Carl Eastlund wrote: On Wed, Jun 29, 2011 at 9:19 PM, Matthew Flatt mfl...@cs.utah.edu wrote: I've pushed a change to Racket's macro system that throws out the syntax-certificate system and adds a syntax-taint system. Syntax taints, like syntax certificates before, are intended to protect macro expansions from abuse. Abuse means using `local-expand' to extract a piece of an expansion, then putting the piece in a new context or using `datum-syntax' to access unexported module bindings using the piece's lexical context. Meanwhile, program-processing tools like `errortrace' or languages like `typed/racket' are supposed to pull apart expansions and reorganize them; code inspectors remain the way to distinguish trusted tools and languages from potential abusers. Things you need to know: * When writing a macro (without `syntax-rules', `define-syntax-rule', or `syntax-id-rules'), apply `syntax-protect' to the macro's result syntax object to protect it from abuse. The `syntax-rules', etc., forms use `syntax-protect' automatically. Does this mean we should be updating all our old macros now, and adding syntax-protect as boilerplate to (nearly) all of our new macros? How often is a manual syntax-protect necessary? It's great to have a system we can reason about, but the nice thing about certificates was that they were invisible most of the time. It looks like you're suggesting that at least the use of syntax-protect is going to become ubiquitous in this new system. That's exactly what I'm worried about too. (I was going to try things out first, but the build crashed.) An attempt to clarify what's unclear to me: if I'm supposed to almost always arm the result of a transformer, then why not have the expander do that for me, and add some new thing so in the exceptional cases I could do (define-syntax (foo stx) (dont-arm ...blah...)) ? (Apologies if this is explained in the guide section.) Actually, the certificate system was good enough at being invisible that explaining the new system in terms of it isn't effective (at least for me) since certificates were this complicated thing that you need to be aware of in some rare cases but 5 minutes after you're done you'd re-forget what you read. (Again, this is all for a subjective value of you, of course.) -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: http://barzilay.org/ Maze is Life! _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] syntax taints instead of syntax certificates
On Wed, Jun 29, 2011 at 10:10 PM, Eli Barzilay e...@barzilay.org wrote: A few seconds ago, Eli Barzilay wrote: 8 minutes ago, Carl Eastlund wrote: On Wed, Jun 29, 2011 at 9:19 PM, Matthew Flatt mfl...@cs.utah.edu wrote: I've pushed a change to Racket's macro system that throws out the syntax-certificate system and adds a syntax-taint system. Syntax taints, like syntax certificates before, are intended to protect macro expansions from abuse. Abuse means using `local-expand' to extract a piece of an expansion, then putting the piece in a new context or using `datum-syntax' to access unexported module bindings using the piece's lexical context. Meanwhile, program-processing tools like `errortrace' or languages like `typed/racket' are supposed to pull apart expansions and reorganize them; code inspectors remain the way to distinguish trusted tools and languages from potential abusers. Things you need to know: * When writing a macro (without `syntax-rules', `define-syntax-rule', or `syntax-id-rules'), apply `syntax-protect' to the macro's result syntax object to protect it from abuse. The `syntax-rules', etc., forms use `syntax-protect' automatically. Does this mean we should be updating all our old macros now, and adding syntax-protect as boilerplate to (nearly) all of our new macros? How often is a manual syntax-protect necessary? It's great to have a system we can reason about, but the nice thing about certificates was that they were invisible most of the time. It looks like you're suggesting that at least the use of syntax-protect is going to become ubiquitous in this new system. That's exactly what I'm worried about too. (I was going to try things out first, but the build crashed.) An attempt to clarify what's unclear to me: if I'm supposed to almost always arm the result of a transformer, then why not have the expander do that for me, and add some new thing so in the exceptional cases I could do (define-syntax (foo stx) (dont-arm ...blah...)) ? (Apologies if this is explained in the guide section.) I was about to ask the same thing. The default seems to be backward, which at first glance doesn't seem hard to fix. Actually, the certificate system was good enough at being invisible that explaining the new system in terms of it isn't effective (at least for me) since certificates were this complicated thing that you need to be aware of in some rare cases but 5 minutes after you're done you'd re-forget what you read. (Again, this is all for a subjective value of you, of course.) _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] syntax taints instead of syntax certificates
On Wed, Jun 29, 2011 at 9:19 PM, Matthew Flatt mfl...@cs.utah.edu wrote: * If you write program-processing tools or languages that use `syntax-recertify', you'll need to change them to use `syntax-disarm' and possibly `syntax-rearm'. Does this change (more generally than the part I've quoted here) mean that code that uses `syntax-recertify' and related functions will no longer compile and/or work in new versions of Racket? -- sam th sa...@ccs.neu.edu _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] syntax taints instead of syntax certificates
8 minutes ago, Carl Eastlund wrote: On Wed, Jun 29, 2011 at 9:19 PM, Matthew Flatt mfl...@cs.utah.edu wrote: Things you need to know: * When writing a macro (without `syntax-rules', `define-syntax-rule', or `syntax-id-rules'), apply `syntax-protect' to the macro's result syntax object to protect it from abuse. The `syntax-rules', etc., forms use `syntax-protect' automatically. Does this mean we should be updating all our old macros now, and adding syntax-protect as boilerplate to (nearly) all of our new macros? How often is a manual syntax-protect necessary? I think you want to use `syntax-protect' on any exported macro. It only actually matters if some untrusted code is going to use a trusted instance of your library. Macros exported by `racket' are surely in that category, but maybe it doesn't matter for most libraries. It's great to have a system we can reason about, but the nice thing about certificates was that they were invisible most of the time. It looks like you're suggesting that at least the use of syntax-protect is going to become ubiquitous in this new system. Having to decorate trustworthy libraries with `syntax-protect' is definitely a drawback. On Wed, Jun 29, 2011 at 10:10 PM, Eli Barzilay e...@barzilay.org wrote: An attempt to clarify what's unclear to me: if I'm supposed to almost always arm the result of a transformer, then why not have the expander do that for me, and add some new thing so in the exceptional cases I could do (define-syntax (foo stx) (dont-arm ...blah...)) ? Sorry --- that point was buried too deeply in my message (first sub-bullet for the last bullet in the third set of bullets). The problem with having the macro transformer add `syntax-protect' for (define-syntax bar (lambda (stx) ...)) is that some other transformer can say ((syntax-local-value #'bar) stx) to get the transformer's result without `syntax-protect' --- which was a gaping hole that Ryan noticed in the certificate system. We considered ways of automating `syntax-protect' for all macros, but the ways we found seemed to create more problems than they solved. _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] syntax taints instead of syntax certificates
On Wed, Jun 29, 2011 at 10:55 PM, Matthew Flatt mfl...@cs.utah.edu wrote: 8 minutes ago, Carl Eastlund wrote: On Wed, Jun 29, 2011 at 9:19 PM, Matthew Flatt mfl...@cs.utah.edu wrote: Things you need to know: * When writing a macro (without `syntax-rules', `define-syntax-rule', or `syntax-id-rules'), apply `syntax-protect' to the macro's result syntax object to protect it from abuse. The `syntax-rules', etc., forms use `syntax-protect' automatically. Does this mean we should be updating all our old macros now, and adding syntax-protect as boilerplate to (nearly) all of our new macros? How often is a manual syntax-protect necessary? I think you want to use `syntax-protect' on any exported macro. It only actually matters if some untrusted code is going to use a trusted instance of your library. Macros exported by `racket' are surely in that category, but maybe it doesn't matter for most libraries. It's great to have a system we can reason about, but the nice thing about certificates was that they were invisible most of the time. It looks like you're suggesting that at least the use of syntax-protect is going to become ubiquitous in this new system. Having to decorate trustworthy libraries with `syntax-protect' is definitely a drawback. On Wed, Jun 29, 2011 at 10:10 PM, Eli Barzilay e...@barzilay.org wrote: An attempt to clarify what's unclear to me: if I'm supposed to almost always arm the result of a transformer, then why not have the expander do that for me, and add some new thing so in the exceptional cases I could do (define-syntax (foo stx) (dont-arm ...blah...)) ? Sorry --- that point was buried too deeply in my message (first sub-bullet for the last bullet in the third set of bullets). The problem with having the macro transformer add `syntax-protect' for (define-syntax bar (lambda (stx) ...)) is that some other transformer can say ((syntax-local-value #'bar) stx) to get the transformer's result without `syntax-protect' --- which was a gaping hole that Ryan noticed in the certificate system. We considered ways of automating `syntax-protect' for all macros, but the ways we found seemed to create more problems than they solved. Thank you, that greatly clarifies the issue for me. Of course I would like to see syntax-case and syntax-parse have options to automatically syntax-protect as well; that will cover a lot more common cases for me. But now I can see where the trade-offs are that resulted in this design. --Carl _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
