Interesting ideas here, Martin, thanks! I'm really not attached to JSON, so don't worry on that account.
I won't have any time to work on my parser for the next couple of weeks, but will consider these things when I start again after that. cheers, Simon On Sun, 19 Nov 2023, 9:02 am Martin Blais, <[email protected]> wrote: > On Sat, Nov 18, 2023 at 2:53 PM Simon Guest <[email protected]> wrote: > >> I had seen that, and it looks interesting! >> >> How could we free it from being C++ only? If such a test suite could be >> language independent, that would really open things up for testing of other >> language implementations. >> > > Well it's based on the original Python one: > > https://github.com/beancount/beancount/blob/master/beancount/parser/parser_test.py > > If you have Python bindings and rendering woud be sufficient to get this > right, you'd use the Python bindings to run the tests > I think this parser_test.py could be converted to assert the data > structure as the C++ one does. > The elegant thing about the C++ one is that since it emits protobufs, > using ascii protobuf syntax naturally provides a way to describe the data > structure. > Do the Rust structs have an elegant ser/de to/from a text syntax? > > > The only thought I had so far would be for a parser adapter layer to >> output as, say JSON, which could be used for comparing with expected output >> for such a suite. >> > > Urg... JSON. > I know JSON's popular, but it would make a rather crappy syntax for the > complexity of this schema. > (I know that's really widespread in the OSS community, and that's a real > shame. JSON sucks with its three types. > protos much nicer and better defined. See the C++ test strings...) > > I think the right thing to do would be to make the Rust parser produce > protos and then you can create/assert equality against them in any language > (e.g. Python of the Rust produced data). Protos are nice in that way, > they're language independent. > > But then we'd back to square one: how do you pass protos back and forth > btw Rust and Python without making copies. > And besides, I'm pretty sure you'll prefer the Rust data structures over > protos. > So maybe implement an elegant ser/de to some ascii format to/from Rust > data structures, as you would with protos. > Then create a binding from Python to accept and parse a string that > creates the corresponding Rust data structures. > Then implement a comparator function. > That's all a bunch of work -- none of it is rocket science, but you're > really just redoing protos at that point. > > > > >> cheers, >> Simon >> >> On Sun, 19 Nov 2023, 7:16 am Martin Blais, <[email protected]> wrote: >> >>> Have you guys seen this suite of tests? >>> >>> https://github.com/beancount/beancount/blob/master/beancount/cparser/parser_test.cc >>> >>> Any parser that supports a suite like this I'd probably want to >>> integrate and contemplate a Rust version of Beancount. >>> There are two crucial things at the bottom of it all: >>> - a good parser >>> - the ability to call to & from Python without any copies >>> Anything else is a piece of cake. >>> >>> >>> >>> On Wed, Nov 15, 2023 at 5:17 PM Simon Guest <[email protected]> >>> wrote: >>> >>>> beancount-parser-lima may do a bit more than the existing core parser >>>> in Beancount proper. It processes all the included files and returns a >>>> date-ordered list of all the directives, with all pragmas either processed >>>> or returned as a single Options structure. >>>> >>>> It doesn't do the filling in of missing numbers, because that seems >>>> tricky indeed (and an application level thing not a parser level thing). >>>> Nor plugins. >>>> >>>> I'm currently making good progress with Python bindings, including some >>>> performance optimisations to reduce the number of allocations of Python >>>> objects. >>>> >>>> On Thursday, 16 November 2023 at 00:29:01 UTC+13 [email protected] wrote: >>>> >>>>> The c++ parser in V3 is pretty much done. >>>>> This is other people writing a parser in rust. >>>>> >>>>> It's worth noting that n beancount the parser alone isn't that useful. >>>>> It's because beancount processes its data in two stages: parsing, which >>>>> produces roughly a data structure that matches the input, and then running >>>>> over the stream of transactions to fill in missing numbers and also >>>>> running >>>>> all the plugins. The result of that is the final stream of transactions >>>>> that you can do queries on. The logic in the second part is where all the >>>>> complexity lies. I haven't ported that. >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Nov 15, 2023, 04:54 Chary Chary <[email protected]> wrote: >>>>> >>>>>> Martin, >>>>>> >>>>>> is this somehow alternative to the C++ parser, which you are working >>>>>> on in beancount v3? >>>>>> >>>>>> On Sunday, November 12, 2023 at 9:46:50 PM UTC+1 [email protected] >>>>>> wrote: >>>>>> >>>>>>> Thank you for sharing. Added to contrib doc. >>>>>>> >>>>>>> >>>>>>> On Sun, Nov 12, 2023 at 3:15 PM Simon Guest <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Ahoy Beancounters! >>>>>>>> >>>>>>>> Of interest really only to developers, I created a new parser for >>>>>>>> Beancount in Rust, called beancount-parser-lima >>>>>>>> <https://docs.rs/beancount-parser-lima/latest/beancount_parser_lima/> >>>>>>>> (because >>>>>>>> I am aware it is not the first such). >>>>>>>> >>>>>>>> I'm currently actively working on this, with my current focus being >>>>>>>> adding Python bindings. >>>>>>>> >>>>>>>> So far it parses everything I have tried. No support for plugins, >>>>>>>> because, well, it's purely Rust so far! (Also no support so far for >>>>>>>> query >>>>>>>> and custom directives, unsure how important they are.) >>>>>>>> >>>>>>>> Performance and rich error reporting are the two headlines. >>>>>>>> >>>>>>>> Hopefully this may be useful for some. >>>>>>>> >>>>>>>> cheers, >>>>>>>> Simon >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Beancount" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/beancount/d63af5c5-2670-4a65-9a88-570dabbb19fbn%40googlegroups.com >>>>>>>> <https://groups.google.com/d/msgid/beancount/d63af5c5-2670-4a65-9a88-570dabbb19fbn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Beancount" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> >>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/beancount/c82304a1-125a-4f93-a149-5717ce010079n%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/beancount/c82304a1-125a-4f93-a149-5717ce010079n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Beancount" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/beancount/f90b32ee-8226-46c9-831d-17450ac831e9n%40googlegroups.com >>>> <https://groups.google.com/d/msgid/beancount/f90b32ee-8226-46c9-831d-17450ac831e9n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Beancount" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/beancount/CAK21%2BhOciLhn-b%2BjBTxdp622BnWhyEtTjLQYRo6Adj5bzZvNkA%40mail.gmail.com >>> <https://groups.google.com/d/msgid/beancount/CAK21%2BhOciLhn-b%2BjBTxdp622BnWhyEtTjLQYRo6Adj5bzZvNkA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Beancount" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/beancount/CAFhGSbvD1XjHLXyjqnrFLst1tyVCUF7Q-XBgzBc8eSxwTpVL%2BQ%40mail.gmail.com >> <https://groups.google.com/d/msgid/beancount/CAFhGSbvD1XjHLXyjqnrFLst1tyVCUF7Q-XBgzBc8eSxwTpVL%2BQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "Beancount" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/beancount/CAK21%2BhNWUcsMYWMzAjiuwRHjNF6kXsRmREDXAHZRweoMCSSHkA%40mail.gmail.com > <https://groups.google.com/d/msgid/beancount/CAK21%2BhNWUcsMYWMzAjiuwRHjNF6kXsRmREDXAHZRweoMCSSHkA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Beancount" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAFhGSbsWHATs%3DhWFBJy_7qD96X5g6XF_wd%3D0UXdMiJgJ25TAEg%40mail.gmail.com.
