Re: [go-nuts] GoAWK: an AWK interpreter written in Go

[Ben Hoyt]

> I wonder if it makes sense, to expose more of the interpreter to go.
> E.g.: register a user function or add an action written in go.
>

I had thought about this before, but your comment made me want to try it.
It wasn't actually that hard, so I've added backwards-compatible support
for this now (GoAWK v1.1.0). It works kind of like Funcs() in text/template
or html/template.

You can pass in your Go functions as a map[string]interface{}, and as long
as they take and return bools, numbers, or strings (or []byte) it'll use
reflection to do all the hard work for you. It also supports variadic
functions. Functions defined in AWK with "function foo" take precedence
over Go functions passed in via Funcs (I did this so that passing in
different Funcs doesn't change the behavior of certain AWK scripts). See
more docs under the "Funcs" field here:
https://godoc.org/github.com/benhoyt/goawk/interp#Config

It's kind of a solution in search of a problem right now, but you could use
this to call something simple like strings.Repeat, or something complex
like doing an HTTP request. It was also good to learn more about how to use
the "reflect" package.

Again, feedback welcome.

-Ben

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] GoAWK: an AWK interpreter written in Go

[Tong Sun]

On Sat, Nov 24, 2018 at 10:11 AM Ben Hoyt wrote:

For what it's worth, I found that tool quite interesting and relevant.
> Tong, I think you're overreacting a bit here.
>

 Oh, I'm sorry, *everyone*.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] GoAWK: an AWK interpreter written in Go

[Ben Hoyt]

>
> probably this  is better tool for the
>> case
>>
>
> - please don't just post a link, but include descriptions.
> - please justify your claim "better" - a specific log file parser is
> better than general-purpose awk scripts? In what way?
> - please don't hijack a thread in promoting your own tool, especially when
> it is highly irreverent.
>

For what it's worth, I found that tool quite interesting and relevant.
Tong, I think you're overreacting a bit here.

-Ben

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] GoAWK: an AWK interpreter written in Go

[Ben Hoyt]

> Hope there are future plans to extend it so that,
> - people can define user function in go and use it in goawk
> 
>  handling,
> as kty... has pointed out.
>

I toyed with this, and enabling it simple functions like "math.Abs(float64)
float64". You could add math.Abs to a new config.Funcs map and then
reflection would figure out the types at runtime and call it. In AWK you'd
just do "abs(-5)" and it'd return 5.

It's a bit harder with functions that take make complex params and return
an error. What would the API look like on the AWK side? Currently AWK knows
at parse time what kinds (scalar or array) each parameter is. For example,
what about an HTTP request function that took method, url, body and
returned status code, headers, and response body. I was thinking for this
case you could have the concept of "complex function" which always took two
parameters, input args array and output values array. So for http_request
it'd look something like:

in["method"] = "GET"
in["url"] = "https://golang.org/;
err = http_request(in, out)
if (err != "") {
  print "network error fetching page:", err
  exit 1
}
if (out["status"] == 404) {
  print "page not found"
  exit 1
}
print out["body"]

It's a bit klunky but at least the API is consistent: functions always take
input and output arrays and return an error string (or "").

Anyone here have thoughts on a better AWK-level API?

- also enable normal go program to use awk scripts
>

I'm not sure what you mean here. You can already use GoAWK in normal Go
programs against any io.Reader input and io.Writer for output.

-Ben

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] GoAWK: an AWK interpreter written in Go

[Tong Sun]

On Sat, Nov 24, 2018 at 6:44 AM Denis Cheremisov wrote:

> >  - also enable normal go program to use awk scripts
>
> probably this  is better tool for the
> case
>

- please don't just post a link, but include descriptions.
- please justify your claim "better" - a specific log file parser is better
than general-purpose awk scripts? In what way?
- please don't hijack a thread in promoting your own tool, especially when
it is highly irreverent.

Thank you for your cooperation

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] GoAWK: an AWK interpreter written in Go

[Denis Cheremisov]

>  - also enable normal go program to use awk scripts

probably this  is better tool for the 
case

суббота, 24 ноября 2018 г., 5:43:22 UTC+3 пользователь Tong Sun написал:
>
>
>
> On Saturday, November 17, 2018 at 12:44:57 PM UTC-5, Ben Hoyt wrote:
>>
>>
>> https://benhoyt.com/writings/goawk/ 
>> 
>>
>> It's pretty much a toy project, not being used for anything real, but it 
>> seems to be in a good shape. Feedback welcome!
>>
>
> Hope there are future plans to extend it so that, 
>
> - people can define user function in go and use it in goawk 
> 
>  handling, 
> as kty... has pointed out. 
> - also enable normal go program to use awk scripts
>
> thx
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] GoAWK: an AWK interpreter written in Go

[Tong Sun]

On Saturday, November 17, 2018 at 12:44:57 PM UTC-5, Ben Hoyt wrote:
>
>
> https://benhoyt.com/writings/goawk/ 
> 
>
> It's pretty much a toy project, not being used for anything real, but it 
> seems to be in a good shape. Feedback welcome!
>

Hope there are future plans to extend it so that, 

- people can define user function in go and use it in goawk 

 handling, 
as kty... has pointed out. 
- also enable normal go program to use awk scripts

thx

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] GoAWK: an AWK interpreter written in Go

[Ben Hoyt]

Thank you. I knew Weinberger was at Google, but didn't know he was on the
Go team -- cool!

On Sat, Nov 17, 2018 at 6:03 PM Alan Donovan  wrote:

> This is great, both as an "étude"---a challenge for sharpening your
> technique---and as an exemplary write-up of the process of building
> something non-trivial and making it both correct and fast. Nice work.
>
> I'm sure you knew already, but Peter Weinberger (the W in AWK) is on the
> Go team at Google.
>
>
>
> On 17 November 2018 at 12:44, Ben Hoyt  wrote:
>
>> I've "finished" my Go AWK interpreter and released v1.0.0 now. I've fixed
>> several bugs, and also sped up the interpreter significantly, primarily by:
>>
>> 1) Resolving variable names to integers at parse time so we can do
>> []value lookups instead of map[string]value lookups at runtime
>> 2) Using normal error returns instead of panic/recover internally for
>> interpreter errors (but oh for the proposed "check" keyword to make this
>> less verbose!)
>> 3) Using switch/case to switch on binary operation type instead of
>> looking up the operation in a map of functions
>> 4) Avoiding allocations in many instances
>>
>> Here's a write-up about GoAWK for those that are interested:
>>
>> https://benhoyt.com/writings/goawk/
>>
>> It's pretty much a toy project, not being used for anything real, but it
>> seems to be in a good shape. Feedback welcome!
>>
>> -Ben
>>
>>
>> On Fri, Aug 24, 2018 at 5:13 PM Ben Hoyt  wrote:
>>
>>> I recently wrote an AWK interpreter in Go:
>>> https://github.com/benhoyt/goawk
>>>
>>> It's a pretty simple implementation: hand-rolled lexer,
>>> recursive-descent parser, and tree-walk interpreter. It's pretty complete
>>> according to the POSIX spec, and it passes the AWK ("one true awk") test
>>> suite as well as my own unit tests.
>>>
>>> In some quick tests I ran, I/O speed is on a par or better than AWK but
>>> the interpreter itself is quite slow -- about 5x slower for a lot of
>>> things. I hope to add some proper benchmarks soon. I have a pretty good of
>>> why and how to fix it: variable lookup and assignment is slow, and I'm
>>> planning to fix by resolving more things at parse time.
>>>
>>> One thing that's a bit funky about AWK is its type handling: string
>>> values can be real strings or "numeric strings" (numbers that came from
>>> user input). I'm currently passing the "value" struct (see interp/value.go)
>>> around by value. I still need to test if that's a good idea
>>> performance-wise or not.
>>>
>>> I'd love to hear any comments, bug reports, or -- especially -- code
>>> reviews.
>>>
>>> -Ben
>>>
>>> --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "golang-nuts" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/golang-nuts/kYZp3Q1KKfE/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> golang-nuts+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] GoAWK: an AWK interpreter written in Go

['Alan Donovan' via golang-nuts]

This is great, both as an "étude"---a challenge for sharpening your
technique---and as an exemplary write-up of the process of building
something non-trivial and making it both correct and fast. Nice work.

I'm sure you knew already, but Peter Weinberger (the W in AWK) is on the Go
team at Google.



On 17 November 2018 at 12:44, Ben Hoyt  wrote:

> I've "finished" my Go AWK interpreter and released v1.0.0 now. I've fixed
> several bugs, and also sped up the interpreter significantly, primarily by:
>
> 1) Resolving variable names to integers at parse time so we can do []value
> lookups instead of map[string]value lookups at runtime
> 2) Using normal error returns instead of panic/recover internally for
> interpreter errors (but oh for the proposed "check" keyword to make this
> less verbose!)
> 3) Using switch/case to switch on binary operation type instead of looking
> up the operation in a map of functions
> 4) Avoiding allocations in many instances
>
> Here's a write-up about GoAWK for those that are interested:
>
> https://benhoyt.com/writings/goawk/
>
> It's pretty much a toy project, not being used for anything real, but it
> seems to be in a good shape. Feedback welcome!
>
> -Ben
>
>
> On Fri, Aug 24, 2018 at 5:13 PM Ben Hoyt  wrote:
>
>> I recently wrote an AWK interpreter in Go: https://github.com/benhoyt/
>> goawk
>>
>> It's a pretty simple implementation: hand-rolled lexer, recursive-descent
>> parser, and tree-walk interpreter. It's pretty complete according to the
>> POSIX spec, and it passes the AWK ("one true awk") test suite as well as my
>> own unit tests.
>>
>> In some quick tests I ran, I/O speed is on a par or better than AWK but
>> the interpreter itself is quite slow -- about 5x slower for a lot of
>> things. I hope to add some proper benchmarks soon. I have a pretty good of
>> why and how to fix it: variable lookup and assignment is slow, and I'm
>> planning to fix by resolving more things at parse time.
>>
>> One thing that's a bit funky about AWK is its type handling: string
>> values can be real strings or "numeric strings" (numbers that came from
>> user input). I'm currently passing the "value" struct (see interp/value.go)
>> around by value. I still need to test if that's a good idea
>> performance-wise or not.
>>
>> I'd love to hear any comments, bug reports, or -- especially -- code
>> reviews.
>>
>> -Ben
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "golang-nuts" group.
>> To unsubscribe from this topic, visit https://groups.google.com/d/
>> topic/golang-nuts/kYZp3Q1KKfE/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> golang-nuts+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] GoAWK: an AWK interpreter written in Go

[ktye78]

That looks nice!
I wonder if it makes sense, to expose more of the interpreter to go.
E.g.: register a user function or add an action written in go.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] GoAWK: an AWK interpreter written in Go

[Ben Hoyt]

I've "finished" my Go AWK interpreter and released v1.0.0 now. I've fixed
several bugs, and also sped up the interpreter significantly, primarily by:

1) Resolving variable names to integers at parse time so we can do []value
lookups instead of map[string]value lookups at runtime
2) Using normal error returns instead of panic/recover internally for
interpreter errors (but oh for the proposed "check" keyword to make this
less verbose!)
3) Using switch/case to switch on binary operation type instead of looking
up the operation in a map of functions
4) Avoiding allocations in many instances

Here's a write-up about GoAWK for those that are interested:

https://benhoyt.com/writings/goawk/

It's pretty much a toy project, not being used for anything real, but it
seems to be in a good shape. Feedback welcome!

-Ben


On Fri, Aug 24, 2018 at 5:13 PM Ben Hoyt  wrote:

> I recently wrote an AWK interpreter in Go:
> https://github.com/benhoyt/goawk
>
> It's a pretty simple implementation: hand-rolled lexer, recursive-descent
> parser, and tree-walk interpreter. It's pretty complete according to the
> POSIX spec, and it passes the AWK ("one true awk") test suite as well as my
> own unit tests.
>
> In some quick tests I ran, I/O speed is on a par or better than AWK but
> the interpreter itself is quite slow -- about 5x slower for a lot of
> things. I hope to add some proper benchmarks soon. I have a pretty good of
> why and how to fix it: variable lookup and assignment is slow, and I'm
> planning to fix by resolving more things at parse time.
>
> One thing that's a bit funky about AWK is its type handling: string values
> can be real strings or "numeric strings" (numbers that came from user
> input). I'm currently passing the "value" struct (see interp/value.go)
> around by value. I still need to test if that's a good idea
> performance-wise or not.
>
> I'd love to hear any comments, bug reports, or -- especially -- code
> reviews.
>
> -Ben
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/golang-nuts/kYZp3Q1KKfE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.