Re: [go-nuts] Experience Report: Using Coroutines for Parsing

[roger peppe]

On Mon, 18 Nov 2024 at 18:02, Romain Doumenc  wrote:

> Hi Roger,
>
> Glad you found this interesting, and thank you for reading !
>
> One can obviously always use methods on a structure instead of
> coroutines (in the example I provided, the state could be stored in
> the tokenizer), and the question is when storing the state on the Go
> stack directly makes code more readable


That's not quite what I meant. I was suggesting exactly storing the state
on the Go stack, just as you're doing with your coroutines, but instead
of using the internal Go API, ISTM that it should be possible to use
iter.Pull , which uses coroutines under the
hood, instead.
Here is a sketch of the idea: https://go.dev/play/p/pFHvaSgOn9c

  cheers,
rog.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/CAJhgaciiHAH_heceV2R8ozveVzg6GrVdBfrhNoOv2uKdOo8Cew%40mail.gmail.com.

Re: [go-nuts] Experience Report: Using Coroutines for Parsing

[robert engels]

Btw, wasn’t being dismissive of your effort - I only linked to the article 
(which is pretty old) because it covers many of the same ideas you were 
discussing - and whenever I get the chance to link to something involving Knuth 
I am going to take it :)

> On Nov 18, 2024, at 12:02 PM, Romain Doumenc  wrote:
> 
> Hi Roger,
> 
> Glad you found this interesting, and thank you for reading !
> 
> One can obviously always use methods on a structure instead of
> coroutines (in the example I provided, the state could be stored in
> the tokenizer), and the question is when storing the state on the Go
> stack directly makes code more readable – and yes, a lot of what
> “readable” means is colored by one’s own experience.
> I think the largest benefit does not stem from the feedback from the
> parser to the tokeniser (although I do find it very cute), but instead
> from the ability of the parent routine to restart the parser when an
> error is found. With coroutines, this can be expressed as a simple for
> loop; but I have not seen something as neat with other methods –
> either stack unwinding (hard to maintain invariants) or instead
> pushing deep down in the parser the knowledge about how much tokens to
> throw away.
> 
> My goal in reaching code currently hidden in the library was to see
> how much more we could do if we had full coroutines, and not making
> any claim about finding a “best” or even a “better” approach. Looking
> forward to seeing what you create with iterators!
> 
> Best
> --
> Romain
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion visit 
> https://groups.google.com/d/msgid/golang-nuts/CAMYUV_oa_DEWceZUFx6EqPjkJ1cqVA5GS6pQEeK-h%3DkrExAhiA%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/8CCEFA4E-7FA5-4AB2-86A0-00BB3C792299%40ix.netcom.com.

Re: [go-nuts] Experience Report: Using Coroutines for Parsing

[Romain Doumenc]

Hi Roger,

Glad you found this interesting, and thank you for reading !

One can obviously always use methods on a structure instead of
coroutines (in the example I provided, the state could be stored in
the tokenizer), and the question is when storing the state on the Go
stack directly makes code more readable – and yes, a lot of what
“readable” means is colored by one’s own experience.
I think the largest benefit does not stem from the feedback from the
parser to the tokeniser (although I do find it very cute), but instead
from the ability of the parent routine to restart the parser when an
error is found. With coroutines, this can be expressed as a simple for
loop; but I have not seen something as neat with other methods –
either stack unwinding (hard to maintain invariants) or instead
pushing deep down in the parser the knowledge about how much tokens to
throw away.

My goal in reaching code currently hidden in the library was to see
how much more we could do if we had full coroutines, and not making
any claim about finding a “best” or even a “better” approach. Looking
forward to seeing what you create with iterators!

Best
--
Romain

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/CAMYUV_oa_DEWceZUFx6EqPjkJ1cqVA5GS6pQEeK-h%3DkrExAhiA%40mail.gmail.com.

Re: [go-nuts] Experience Report: Using Coroutines for Parsing

[robert engels]

I agree, all you need is generators - that they are implemented with coroutines 
is immaterial.

Take a look at this, and substitute the yield() function for crReturn().

https://www.chiark.greenend.org.uk/~sgtatham/coroutines.html

> On Nov 18, 2024, at 10:54 AM, roger peppe  wrote:
> 
> Interesting! Thanks for sharing.
> 
> This is related to something that I've been planning to explore for a while 
> now: using iterators themselves and their associated coroutines accessed via 
> `iter.Pull`, to do parsing. That is, instead of gaining access to the 
> underlying coroutine machinery by use of "go:linkname", perhaps we can get a 
> bunch of the same advantages by using the more structured form already 
> provided by the standard library.
> 
> I suspect that the feedback from parser to scanner you describe can be 
> implemented by allowing the parser to invoke methods on the iteration values 
> (tokens) returned by the scanner.
> 
> I wonder if the result might actually be nicer (easier to understand and 
> better structured) than the direct use of coroutines, which personally I find 
> quite hard to "get".
> 
>   rog.
> 
> On Mon, 18 Nov 2024 at 02:53, Romain Doumenc  wrote:
>> Hi everyone,
>> 
>> I took some time over the past weeks to experiment with the coroutine 
>> switching capabilities in the runtime – in this case for parsing.
>> I find this new API extremely useful, and started seeing quite a few places 
>> in the code base I maintain to simplify some complicated, synchronized, 
>> stateful machines.
>> 
>> The document is at: 
>> https://trout-software.notion.site/Parsing-with-Coroutines-13bf9f8a093980d7a0fcf41b81594677
>> 
>> I hope it is useful for others also interested in experimenting
>> --
>> Romain
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com 
>> .
>> To view this discussion visit 
>> https://groups.google.com/d/msgid/golang-nuts/d095a5d6-679f-4c13-82d3-7acf07cf9996n%40googlegroups.com
>>  
>> .
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion visit 
> https://groups.google.com/d/msgid/golang-nuts/CAJhgacjwXX6EeRt5%2BzDhy_15wXyvNxHv8CAfLvu0nTz1w2KJPQ%40mail.gmail.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/3AA3318C-2F6E-4683-AE15-14436008F925%40ix.netcom.com.

Re: [go-nuts] Experience Report: Using Coroutines for Parsing

[roger peppe]

Interesting! Thanks for sharing.

This is related to something that I've been planning to explore for a while
now: using iterators themselves and their associated coroutines accessed
via `iter.Pull`, to do parsing. That is, instead of gaining access to the
underlying coroutine machinery by use of "go:linkname", perhaps we can get
a bunch of the same advantages by using the more structured form already
provided by the standard library.

I suspect that the feedback from parser to scanner you describe can be
implemented by allowing the parser to invoke methods on the iteration
values (tokens) returned by the scanner.

I wonder if the result might actually be nicer (easier to understand and
better structured) than the direct use of coroutines, which personally I
find quite hard to "get".

  rog.

On Mon, 18 Nov 2024 at 02:53, Romain Doumenc  wrote:

> Hi everyone,
>
> I took some time over the past weeks to experiment with the coroutine
> switching capabilities in the runtime – in this case for parsing.
> I find this new API extremely useful, and started seeing quite a few
> places in the code base I maintain to simplify some complicated,
> synchronized, stateful machines.
>
> The document is at:
> https://trout-software.notion.site/Parsing-with-Coroutines-13bf9f8a093980d7a0fcf41b81594677
>
> I hope it is useful for others also interested in experimenting
> --
> Romain
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion visit
> https://groups.google.com/d/msgid/golang-nuts/d095a5d6-679f-4c13-82d3-7acf07cf9996n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/CAJhgacjwXX6EeRt5%2BzDhy_15wXyvNxHv8CAfLvu0nTz1w2KJPQ%40mail.gmail.com.