Re: [go-nuts] Preprocessing "go" statements with preserved semantics

[André Eriksson]

On Friday, November 8, 2019 at 10:27:32 PM UTC+1, Ian Lance Taylor wrote:
>
> On Fri, Nov 8, 2019 at 11:40 AM André Eriksson  > wrote: 
> > 
> > On Friday, November 8, 2019 at 7:16:42 PM UTC+1, Ian Lance Taylor wrote: 
> >> 
> >> On Fri, Nov 8, 2019 at 10:06 AM André Eriksson  
> wrote: 
> >> > 
> >> > Interesting. Do you have a reference to where that happens? 
> >> 
> >> The method (*Package).rewriteCall in cmd/cgo/gcc.go.  But more useful 
> >> might be to experiment with some cgo code and build with `go build 
> >> -work` and look in the $WORK directory to see the generated files. 
> >> 
> >> 
> >> > If i understand you correctly, however, it doesn't appear to solve 
> the case where the called function fn lives in a different package, and 
> takes an argument which is a private type. That is: 
> >> > 
> >> > -- a/a.go -- 
> >> > package a 
> >> > 
> >> > type myString string 
> >> > 
> >> > func Fn(str myString) { } 
> >> > 
> >> > -- b/b.go -- 
> >> > package b 
> >> > 
> >> > import "a" 
> >> > 
> >> > func rewrite() { 
> >> > go a.Fn("some string") 
> >> > } 
> >> > 
> >> > In this example knowing the desired type from the function signature 
> does not help, I would think? 
> >> 
> >> That is correct. 
> >> 
> >> Fortunately for a case like that you don't need to use a temporary 
> >> variable at all, since the argument is a constant. 
> >> 
> >> Ian 
> > 
> > 
> > That makes sense. I did some more research and came across another case 
> where even this might not work. 
> > It appears that there is a distinction between an untyped value and an 
> untyped constant. I hadn't appreciated this distinction until now. 
> > 
> > According to the spec (https://golang.org/ref/spec#Comparison_operators), 
> "Comparison operators compare two operands and yield an untyped boolean 
> value." 
> > As a result, it's possible to get untyped, non-constant values from 
> non-constant expressions. 
> > 
> > So we could have fn(a() == b()) passed to fn(b myBool) with no ability 
> to store this argument in a temporary variable without coercing the type to 
> a regular typed bool. 
>
> Technically true but essentially never happens in practice. 
>
> Ian 
>

I suppose that's fair, but it's hard to come up with concrete rules for 
what is permissible given a particular implementation of the rewrite, which 
makes it hard to explain to potential users of the tool.

Switching gears a bit, I guess this might be easier to implement as a 
modification to the compiler? Or can you think of another way of making the 
general case more tractable?

Thanks,
André 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/2817e2e4-cc78-472e-bb06-7b0aa0322bdd%40googlegroups.com.

Re: [go-nuts] Preprocessing "go" statements with preserved semantics

[André Eriksson]

On Friday, November 8, 2019 at 7:16:42 PM UTC+1, Ian Lance Taylor wrote:
>
> On Fri, Nov 8, 2019 at 10:06 AM André Eriksson  > wrote: 
> > 
> > Interesting. Do you have a reference to where that happens? 
>
> The method (*Package).rewriteCall in cmd/cgo/gcc.go.  But more useful 
> might be to experiment with some cgo code and build with `go build 
> -work` and look in the $WORK directory to see the generated files. 
>
>
> > If i understand you correctly, however, it doesn't appear to solve the 
> case where the called function fn lives in a different package, and takes 
> an argument which is a private type. That is: 
> > 
> > -- a/a.go -- 
> > package a 
> > 
> > type myString string 
> > 
> > func Fn(str myString) { } 
> > 
> > -- b/b.go -- 
> > package b 
> > 
> > import "a" 
> > 
> > func rewrite() { 
> > go a.Fn("some string") 
> > } 
> > 
> > In this example knowing the desired type from the function signature 
> does not help, I would think? 
>
> That is correct. 
>
> Fortunately for a case like that you don't need to use a temporary 
> variable at all, since the argument is a constant. 
>
> Ian 
>

That makes sense. I did some more research and came across another case 
where even this might not work.
It appears that there is a distinction between an untyped value and an 
untyped constant. I hadn't appreciated this distinction until now.

According to the spec (https://golang.org/ref/spec#Comparison_operators), 
"Comparison operators compare two operands and yield an untyped boolean 
value."
As a result, it's possible to get untyped, non-constant values from 
non-constant expressions.

So we could have fn(a() == b()) passed to fn(b myBool) with no ability to 
store this argument in a temporary variable without coercing the type to a 
regular typed bool.


>
> > On Friday, November 8, 2019 at 6:58:02 PM UTC+1, Ian Lance Taylor wrote: 
> >> 
> >> On Fri, Nov 8, 2019 at 9:08 AM André Eriksson  
> wrote: 
> >> > 
> >> > That works in simple cases, but does not work when the expression is 
> an untyped constant, like 1 or nil. In the case of 1 the variable will get 
> a concrete type of int, while fn may accept a float32, or even a private 
> type that cannot be named in the current package. 
> >> 
> >> You might want to look at the rewriting that cmd/cgo does, as it 
> >> handles this exact kind of case.  Basically, for untyped constants, 
> >> you know the desired type, because it's in the function signature.  So 
> >> use that. 
> >> 
> >> It does get kind of complicated, though. 
> >> 
> >> Ian 
> >> 
> >> 
> >> > On Friday, November 8, 2019 at 5:51:10 PM UTC+1, Michael Jones wrote: 
> >> >> 
> >> >> If expr was evaluable in the original code then why not rewrite in 
> place after assigning temporaries? 
> >> >> 
> >> >> go fn(e1,e2) 
> >> >> 
> >> >> { 
> >> >> t1,t2 := e1,e2 
> >> >> go func() { 
> >> >>   defer instrument() 
> >> >>   fn(t1,t2) 
> >> >> } 
> >> >> 
> >> >> 
> >> >> On Fri, Nov 8, 2019 at 8:38 AM André Eriksson  
> wrote: 
> >> >>> 
> >> >>> I am working on a type of Go preprocessor that rewrites source code 
> to add additional instrumentation to certain types of statements. 
> >> >>> 
> >> >>> One such statement is the go statement. I would like to instrument 
> the newly created goroutine, injecting some instrumentation code at the 
> start and finish of the goroutine. 
> >> >>> 
> >> >>> In the simple case, the rewrite is straightforward: 
> >> >>> 
> >> >>> go fn() 
> >> >>> 
> >> >>> becomes 
> >> >>> 
> >> >>> go func() { 
> >> >>> defer instrument()() 
> >> >>> fn() 
> >> >>> }() 
> >> >>> 
> >> >>> However this approach does not work when fn takes parameters. 
> >> >>> If we were to rewrite go fn(expr) into the equivalent form above: 
> >> >>> 
> >> >>> go func() { 
> >> >>> defer instrument()() 
> >> >>> fn(expr) 
> >> >>> }() 
> >> >>> 
> >> >>> 
> >> >>> the semantics change, since in the rewrite exp

Re: [go-nuts] Preprocessing "go" statements with preserved semantics

[André Eriksson]

Interesting. Do you have a reference to where that happens?

If i understand you correctly, however, it doesn't appear to solve the case 
where the called function fn lives in a different package, and takes an 
argument which is a private type. That is:

-- a/a.go --
package a

type myString string

func Fn(str myString) { }

-- b/b.go --
package b

import "a"

func rewrite() {
go a.Fn("some string")
}

In this example knowing the desired type from the function signature does 
not help, I would think?

On Friday, November 8, 2019 at 6:58:02 PM UTC+1, Ian Lance Taylor wrote:
>
> On Fri, Nov 8, 2019 at 9:08 AM André Eriksson  > wrote: 
> > 
> > That works in simple cases, but does not work when the expression is an 
> untyped constant, like 1 or nil. In the case of 1 the variable will get a 
> concrete type of int, while fn may accept a float32, or even a private type 
> that cannot be named in the current package. 
>
> You might want to look at the rewriting that cmd/cgo does, as it 
> handles this exact kind of case.  Basically, for untyped constants, 
> you know the desired type, because it's in the function signature.  So 
> use that. 
>
> It does get kind of complicated, though. 
>
> Ian 
>
>
> > On Friday, November 8, 2019 at 5:51:10 PM UTC+1, Michael Jones wrote: 
> >> 
> >> If expr was evaluable in the original code then why not rewrite in 
> place after assigning temporaries? 
> >> 
> >> go fn(e1,e2) 
> >> 
> >> { 
> >> t1,t2 := e1,e2 
> >> go func() { 
> >>   defer instrument() 
> >>   fn(t1,t2) 
> >> } 
> >> 
> >> 
> >> On Fri, Nov 8, 2019 at 8:38 AM André Eriksson  
> wrote: 
> >>> 
> >>> I am working on a type of Go preprocessor that rewrites source code to 
> add additional instrumentation to certain types of statements. 
> >>> 
> >>> One such statement is the go statement. I would like to instrument the 
> newly created goroutine, injecting some instrumentation code at the start 
> and finish of the goroutine. 
> >>> 
> >>> In the simple case, the rewrite is straightforward: 
> >>> 
> >>> go fn() 
> >>> 
> >>> becomes 
> >>> 
> >>> go func() { 
> >>> defer instrument()() 
> >>> fn() 
> >>> }() 
> >>> 
> >>> However this approach does not work when fn takes parameters. 
> >>> If we were to rewrite go fn(expr) into the equivalent form above: 
> >>> 
> >>> go func() { 
> >>> defer instrument()() 
> >>> fn(expr) 
> >>> }() 
> >>> 
> >>> 
> >>> the semantics change, since in the rewrite expr gets evaluated inside 
> the newly created goroutine, which can change the behavior and introduce 
> data races. 
> >>> 
> >>> My attempts to address this have not been particularly fruitful. 
> >>> 
> >>> One cannot pass in expr as an argument to the closure, because the 
> type of the expression may not have a valid name in the current package 
> (for example if expr evaluates to a private type in some other package). 
> >>> 
> >>> Similarly, if expr is a constant expression (like 1 or nil) the type 
> may depend on the corresponding parameter in fn’s signature. 
> >>> 
> >>> The only semantics-preserving rewrite I can think of revolves around 
> using package reflect, and rewriting like so: 
> >>> 
> >>> go func(fn reflect.Value, vals …reflect.Value) { 
> >>> defer instrument() 
> >>> fn.Call(vals) 
> >>> }(reflect.ValueOf(fn), reflect.ValueOf(expr)) 
> >>> 
> >>> As far as I understand, this should be semantics-preserving, although 
> with a slight performance cost. (Though I imagine the cost of a 
> reflection-based call is dwarfed by the cost of spawning a goroutine.) 
> >>> 
> >>> Unfortunately this also comes with a major downside: the rewritten 
> code does not typecheck identically to the original code. Ideally I would 
> like the rewritten form to cause identical typechecking failures to the old 
> code, so that these errors are caught at compile time without requiring a 
> separate typechecking pass for the original code. 
> >>> 
> >>> Am I correct in the above reasoning? Can anyone think of a way to do 
> this sort of rewrite in a semantics-preserving and typechecking-preserving 
> way? 
> >>> 
> >>> -- 
> >>> You received this message because you are 

Re: [go-nuts] Preprocessing "go" statements with preserved semantics

[André Eriksson]

That works in simple cases, but does not work when the expression is an 
untyped constant, like 1 or nil. In the case of 1 the variable will get a 
concrete type of int, while fn may accept a float32, or even a private type 
that cannot be named in the current package.

On Friday, November 8, 2019 at 5:51:10 PM UTC+1, Michael Jones wrote:
>
> If expr was evaluable in the original code then why not rewrite in place 
> after assigning temporaries?
>
> go fn(e1,e2)
>
> {
> t1,t2 := e1,e2
> go func() { 
>   defer instrument()
>   fn(t1,t2)
> }
>
>
> On Fri, Nov 8, 2019 at 8:38 AM André Eriksson  > wrote:
>
>> I am working on a type of Go preprocessor that rewrites source code to 
>> add additional instrumentation to certain types of statements.
>>
>> One such statement is the go statement. I would like to instrument the 
>> newly created goroutine, injecting some instrumentation code at the start 
>> and finish of the goroutine.
>>
>> In the simple case, the rewrite is straightforward:
>>
>> go fn()
>>
>> becomes
>>
>> go func() {
>> defer instrument()()
>> fn()
>> }()
>>
>> However this approach does not work when fn takes parameters.
>> If we were to rewrite go fn(expr) into the equivalent form above:
>>
>> go func() {
>> defer instrument()()
>> fn(expr)
>> }()
>>
>>
>> the semantics change, since in the rewrite expr gets evaluated inside 
>> the newly created goroutine, which can change the behavior and introduce 
>> data races.
>>
>> My attempts to address this have not been particularly fruitful.
>>
>> One cannot pass in expr as an argument to the closure, because the type 
>> of the expression may not have a valid name in the current package (for 
>> example if expr evaluates to a private type in some other package). 
>>
>> Similarly, if expr is a constant expression (like 1 or nil) the type may 
>> depend on the corresponding parameter in fn’s signature.
>>
>> The only semantics-preserving rewrite I can think of revolves around 
>> using package reflect, and rewriting like so:
>>
>> go func(fn reflect.Value, vals …reflect.Value) {
>> defer instrument()
>> fn.Call(vals)
>> }(reflect.ValueOf(fn), reflect.ValueOf(expr))
>>
>> As far as I understand, this should be semantics-preserving, although 
>> with a slight performance cost. (Though I imagine the cost of a 
>> reflection-based call is dwarfed by the cost of spawning a goroutine.)
>>
>> Unfortunately this also comes with a major downside: the rewritten code 
>> does not typecheck identically to the original code. Ideally I would like 
>> the rewritten form to cause identical typechecking failures to the old 
>> code, so that these errors are caught at compile time without requiring a 
>> separate typechecking pass for the original code.
>>
>> Am I correct in the above reasoning? Can anyone think of a way to do this 
>> sort of rewrite in a semantics-preserving and typechecking-preserving way?  
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golan...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/a92641f3-2eda-4d4a-ab02-d2b40e3bde75%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/golang-nuts/a92641f3-2eda-4d4a-ab02-d2b40e3bde75%40googlegroups.com?utm_medium=email_source=footer>
>> .
>>
>
>
> -- 
>
> *Michael T. jonesmichae...@gmail.com *
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/ec41a345-163f-4a8a-a24f-b868def081a0%40googlegroups.com.

[go-nuts] Preprocessing "go" statements with preserved semantics

[André Eriksson]

I am working on a type of Go preprocessor that rewrites source code to add 
additional instrumentation to certain types of statements.

One such statement is the go statement. I would like to instrument the 
newly created goroutine, injecting some instrumentation code at the start 
and finish of the goroutine.

In the simple case, the rewrite is straightforward:

go fn()

becomes

go func() {
defer instrument()()
fn()
}()

However this approach does not work when fn takes parameters.
If we were to rewrite go fn(expr) into the equivalent form above:

go func() {
defer instrument()()
fn(expr)
}()


the semantics change, since in the rewrite expr gets evaluated inside the 
newly created goroutine, which can change the behavior and introduce data 
races.

My attempts to address this have not been particularly fruitful.

One cannot pass in expr as an argument to the closure, because the type of 
the expression may not have a valid name in the current package (for 
example if expr evaluates to a private type in some other package). 

Similarly, if expr is a constant expression (like 1 or nil) the type may 
depend on the corresponding parameter in fn’s signature.

The only semantics-preserving rewrite I can think of revolves around using 
package reflect, and rewriting like so:

go func(fn reflect.Value, vals …reflect.Value) {
defer instrument()
fn.Call(vals)
}(reflect.ValueOf(fn), reflect.ValueOf(expr))

As far as I understand, this should be semantics-preserving, although with 
a slight performance cost. (Though I imagine the cost of a reflection-based 
call is dwarfed by the cost of spawning a goroutine.)

Unfortunately this also comes with a major downside: the rewritten code 
does not typecheck identically to the original code. Ideally I would like 
the rewritten form to cause identical typechecking failures to the old 
code, so that these errors are caught at compile time without requiring a 
separate typechecking pass for the original code.

Am I correct in the above reasoning? Can anyone think of a way to do this 
sort of rewrite in a semantics-preserving and typechecking-preserving way?  

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/a92641f3-2eda-4d4a-ab02-d2b40e3bde75%40googlegroups.com.

Re: [go-nuts] Re: Isn't the use of Context in db/sql non-diomatic?

[André Eriksson]

Daniel,

I agree that multiple queries are commonplace. I was referring to multiple 
transactions within a single request to be rare, and when it happens you 
don't necessarily want to share the same (read-only, isolation level) 
properties.

You do have a point that there is a use case for making multiple queries, 
without using a transaction, that you likely would want to share the same 
properties. However, my understanding is that isolation level is a 
meaningless concept without a transaction, and that leaves only read-only 
as the property of concern (for now?). It's not clear from the 
documentation whether the read-only property actually affects the other 
*sql.DB methods that take contexts; it only refers to BeginContext.

I also don't think it's unwarranted to argue that if you want to 
conveniently ensure that multiple queries share the same properties that 
you can do so with a transaction. I understand that in rare circumstances 
the transaction would come with other, undesirable SQL semantics, but in 
those cases you might as well go through the trouble of dealing with this 
yourself. Either by passing the options to each query yourself, or to 
create some application-specific abstraction that handlesit for you.

On Saturday, December 10, 2016 at 3:31:12 PM UTC+1, Daniel Theophanes wrote:
>
>
> André, 
>
> Thanks for the constructive feedback. I agree the signature you propose 
> would be another way to do this. 
>
> For people who care about these settings, I would expect multiple queries 
> per request is normal and ensuring the props are the same sounds like a 
> benefit, not a detriment. In other words, I think you may be 
> underestimating how tied these are to a request. 
>
> As another way of framing this, what do people have in their context Value 
> whitelist?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Isn't the use of Context in db/sql non-diomatic?

[André Eriksson]

Daniel, I would characterize the use case you describe as a very indirect 
way of setting these options on transactions. That's can be a good thing 
when contexts are already passed around, and coming up with new ways of 
passing such options down through a few layers of code is some amount of 
work. It's certainly convenient that they are part of the context in that 
case.

That said, I believe there are large downsides to this indirect API design. 
Even when you benefit from this convenience, it encourages a form of 
"spooky action at a distance" programming where it's not clear at the 
BeginContext call site what the properties of the transaction will be. 
Using Go has made me very appreciative of other language features that 
directly remove this "spooky action at a distance" from other programming 
languages. Not having exceptions is a great example of this. I would say 
that Go trades convenience in favor of clarity in those circumstances, and 
I'm concerned that we're making the opposite tradeoff here.

One can then easily argue that nothing is stopping me or anybody else from 
forgoing the indirect nature and finding a way of propagating these options 
down to the code that actually calls BeginContext. In fact, that's probably 
what I would do. The problem then is that the API is needlessly awkward; it 
would be much more straightforward to expose an API with functional options 
if that's the encouraged way of using it.

Another worry of mine is that contexts are passed on to other functions. 
Even though it's rare that a single request consists of multiple 
transactions, it still happens and I wouldn't be too surprised to find 
people accidentally setting the wrong transaction properties simply because 
it was accidentally inherited from the context that was used to begin 
another transaction. Again, being more explicit in the API design would 
cause people to stop and think before passing along a set of transaction 
options from an unrelated transaction, similar to how Go's explicit error 
handling forces you to think about errors, even if it's less convenient.

To end with a concrete proposal, I think changing the signature to be "func 
(*DB) BeginContext(ctx context.Context, opts ...TxOption)" or similar (and 
then changing the transaction properties to be TxOptions) would solve most 
of the issues I highlighted above.

André

On Friday, December 9, 2016 at 9:14:08 PM UTC+1, Daniel Theophanes wrote:
>
> Hi Chandra,
>
> In my view using context to store value for these uses (Read-Only and 
> Isolation Level) is somewhat of a gray area. They are not a classical 
> request only scope, like a tracing ID, but they also aren't strictly 
> limited to a single request either. Say you wanted to kick off a request, 
> but ensure you don't modify the database in anyway. you could pass in a 
> context with ReadOnly set and any further request function will follow that 
> directive. Similarly, by allowing you to set the Isolation at a higher 
> level, you can effectively influence the behavior of the queries for any 
> subsequent request action.
>
> Should we do it this way? Maybe, maybe not. I think there are benefits to 
> this approach, and I think they can be considered request scoped, even if 
> in the degenerate case they are set at the query level. However, I think it 
> will be really common to set these when dispatching requests in the router 
> and using them for the subsequent requests.
>
> Has /report/ prefix? Set to ReadOnly and Isolation X
> Has /api/ prefix? Set Isolation to Y
>
> At least that is how I would envision using them. I have been known to be 
> wrong before.
> What do you think? -Daniel
>
>
> On Thursday, December 8, 2016 at 11:27:36 PM UTC-8, Chandra Sekar S wrote:
>>
>> Bump
>>
>> --
>> Chandra Sekar.S
>>
>> On Wed, Dec 7, 2016 at 10:24 AM, Chandru  wrote:
>>
>>> I can understand db/sql using Context for cancellation. It is the 
>>> optional arguments to BeginContext like IsolationLevel and read-only flag, 
>>> which are not request-specific, that seem to contradict context's 
>>> documentation.
>>>
>>> --
>>> Chandra Sekar.S
>>>
>>> On Tue, Dec 6, 2016 at 9:50 PM,  wrote:
>>>
 Either the doc should be changed or the std lib should follow the 
 spirit of the doc. But my understanding is that context.Context is not 
 just 
 about HTTP request/response cycle but deals with anything that needs 
 cancellation or a resource clean up signal , some kind of DIY RAII . In 
 any 
 case the documentation should be clarified. 

 Le mardi 6 décembre 2016 16:48:48 UTC+1, Chandra Sekar S a écrit :
>
> Documentation of the context package says,
>
> "Use context Values only for request-scoped data that transits 
> processes and APIs, not for passing optional parameters to functions."
>
> sql.BeginContext introduced in 1.8, uses Context to receive options 
> like IsolationLevel and