Re: [capnproto] Incomplete compile-time Zig implementation
Nah, compile-time filesystem access is very limited. The Zig compiler only has a single builtin function for it: @embedFile(), and it's limited to only reading files inside the project directory. Automatically resolving capnp import statements might be possible in some cases, but path resolution will be tricky to get right since @embedFile() resolves relative to the source file calling it. My idea for a workaround was to have the library user do all the @embedFile() calls and pass a list of paths and contents to the parser. On 2023-02-01, 'Kenton Varda' via Cap'n Proto wrote: > Hmm, it's interesting that you're allowed to read non-zig source files from > disk in "comptime" but not allowed to execute external binaries or call > into non-zig libraries. If the capnp file contains an import statement, are > you allowed to open whatever file it references? If so that sounds like you > basically have arbitrary (read-only?) filesystem access? > > -Kenton > > On Tue, Jan 31, 2023 at 2:18 AM Yorhel wrote: > > > Hi Kenton, > > > > The project comes with a whole new parser and layouting implementation. > > I have some tests[1] for complicated struct/union layouts, but you're > > absolutely right that this needs much more rigorous testing and an > > enormous disclaimer - both of which I had planned had I continued. > > > > It's not possible to invoke external programs or call into other > > languages from Zig comptime[2], that would require a separate build > > step. One idea I considered is to have the parser recognize the > > comments generated by `capnp -o capnp` and use them for validation, so > > that it's at least easier to get that extra assurance. Loading a file in > > schema.capnp format could also be done, of course, but then one might as > > well introduce a separate build step. > > > > 1) > > https://code.blicky.net/yorhel/capnzig/src/commit/5618819e01c8dda38962b986e4107d7d20e863f5/src/schema.zig#L812 > > > > 2) Okay, Zig's comptime being turing complete one could potentially > > write a C++ interpreter and run capnp's layouting code during Zig > > compilation, but let's not go there. :) > > > > Yorhel. > > > > On 2023-01-30, 'Kenton Varda' via Cap'n Proto wrote: > > > Hi Yorhel, > > > > > > Cool stuff. It's neat that Zig can run code at compile time. > > > > > > Question, though: are you calling the C++ parser library to parse Cap'n > > > Proto schema language, or did you write a whole new parser for it? If the > > > latter, I'm worried as the algorithm to decide the layout of structs is > > > quite complex, especially when unions of groups are involved. If you are > > > re-implementing that, it would be pretty easy to get wrong leading to > > > subtle incompatibilities. Is it possible to invoke the parser from the > > C++ > > > implementation as a library, to convert from capnp IDL into schema.capnp > > > format, which could then be interpreted by your implementation? > > > > > > -Kenton > > > > > > On Thu, Jan 12, 2023 at 6:06 AM 'Yorhel' via Cap'n Proto < > > > capnproto@googlegroups.com> wrote: > > > > > > > Hi list, > > > > > > > > I've been working on a Cap'n Proto implementation for the Zig language. > > > > This implementation is unusual in that it leverages the language's > > > > compile-time code execution capabilities to read a schema file - > > written > > > > in Cap'n Proto's schema language, no separate compilation step required > > > > - and then make the interfaces defined in the schema available to > > > > library users. > > > > > > > > I wasn't 100% sure if this was going to work, but I now have a parser, > > > > struct layouting algorithm and some rudimentary experiments with a > > > > type-safe struct read/write API that compiles down to code that > > *should* > > > > be just as efficient as if a code generation step was involved. > > > > > > > > Sadly, now that I've convinced myself that this might actually work > > out, > > > > I've lost the motivation to do the remaining 80% of the work and turn > > it > > > > into a usable library. So instead I'm publishing the unfinished product > > > > with the hope that it might inspire or prove useful to someone. And, > > who > > > > knows, maybe I'll continue to work on it some time in the future. > > > > > > > > The code is at https://code.blicky.net/yorhel/capnzig > > > > > > > > Documentation's kind of lacking - the code is commented here and there, > > > > but making sense of everything may take effort. I'll be around to > > answer > > > > questions if there are any. > > > > > > > > Here's hoping we'll have a mature Cap'n Proto implementation for Zig at > > > > some point, regardless of which code generation strategy is taken. > > > > > > > > Yorhel. > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > Groups > > > > "Cap'n Proto" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > an > > > > email to capnproto+unsubscr...@googlegroups.com. > > > >
Re: [capnproto] Incomplete compile-time Zig implementation
Hmm, it's interesting that you're allowed to read non-zig source files from disk in "comptime" but not allowed to execute external binaries or call into non-zig libraries. If the capnp file contains an import statement, are you allowed to open whatever file it references? If so that sounds like you basically have arbitrary (read-only?) filesystem access? -Kenton On Tue, Jan 31, 2023 at 2:18 AM Yorhel wrote: > Hi Kenton, > > The project comes with a whole new parser and layouting implementation. > I have some tests[1] for complicated struct/union layouts, but you're > absolutely right that this needs much more rigorous testing and an > enormous disclaimer - both of which I had planned had I continued. > > It's not possible to invoke external programs or call into other > languages from Zig comptime[2], that would require a separate build > step. One idea I considered is to have the parser recognize the > comments generated by `capnp -o capnp` and use them for validation, so > that it's at least easier to get that extra assurance. Loading a file in > schema.capnp format could also be done, of course, but then one might as > well introduce a separate build step. > > 1) > https://code.blicky.net/yorhel/capnzig/src/commit/5618819e01c8dda38962b986e4107d7d20e863f5/src/schema.zig#L812 > > 2) Okay, Zig's comptime being turing complete one could potentially > write a C++ interpreter and run capnp's layouting code during Zig > compilation, but let's not go there. :) > > Yorhel. > > On 2023-01-30, 'Kenton Varda' via Cap'n Proto wrote: > > Hi Yorhel, > > > > Cool stuff. It's neat that Zig can run code at compile time. > > > > Question, though: are you calling the C++ parser library to parse Cap'n > > Proto schema language, or did you write a whole new parser for it? If the > > latter, I'm worried as the algorithm to decide the layout of structs is > > quite complex, especially when unions of groups are involved. If you are > > re-implementing that, it would be pretty easy to get wrong leading to > > subtle incompatibilities. Is it possible to invoke the parser from the > C++ > > implementation as a library, to convert from capnp IDL into schema.capnp > > format, which could then be interpreted by your implementation? > > > > -Kenton > > > > On Thu, Jan 12, 2023 at 6:06 AM 'Yorhel' via Cap'n Proto < > > capnproto@googlegroups.com> wrote: > > > > > Hi list, > > > > > > I've been working on a Cap'n Proto implementation for the Zig language. > > > This implementation is unusual in that it leverages the language's > > > compile-time code execution capabilities to read a schema file - > written > > > in Cap'n Proto's schema language, no separate compilation step required > > > - and then make the interfaces defined in the schema available to > > > library users. > > > > > > I wasn't 100% sure if this was going to work, but I now have a parser, > > > struct layouting algorithm and some rudimentary experiments with a > > > type-safe struct read/write API that compiles down to code that > *should* > > > be just as efficient as if a code generation step was involved. > > > > > > Sadly, now that I've convinced myself that this might actually work > out, > > > I've lost the motivation to do the remaining 80% of the work and turn > it > > > into a usable library. So instead I'm publishing the unfinished product > > > with the hope that it might inspire or prove useful to someone. And, > who > > > knows, maybe I'll continue to work on it some time in the future. > > > > > > The code is at https://code.blicky.net/yorhel/capnzig > > > > > > Documentation's kind of lacking - the code is commented here and there, > > > but making sense of everything may take effort. I'll be around to > answer > > > questions if there are any. > > > > > > Here's hoping we'll have a mature Cap'n Proto implementation for Zig at > > > some point, regardless of which code generation strategy is taken. > > > > > > Yorhel. > > > > > > -- > > > You received this message because you are subscribed to the Google > Groups > > > "Cap'n Proto" group. > > > To unsubscribe from this group and stop receiving emails from it, send > an > > > email to capnproto+unsubscr...@googlegroups.com. > > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/capnproto/Y7/3v1VZfkFIFO5s%40gmai021 > . > > > > > > > -- > > You received this message because you are subscribed to the Google > Groups "Cap'n Proto" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to capnproto+unsubscr...@googlegroups.com. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DFvmUpqCvqEtGAvU4gtMo5V8drrGcPHXWRuPBYc254PA%40mail.gmail.com > . > -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscr...@googlegroups.com. To view this
Re: [capnproto] Incomplete compile-time Zig implementation
Hi Kenton, The project comes with a whole new parser and layouting implementation. I have some tests[1] for complicated struct/union layouts, but you're absolutely right that this needs much more rigorous testing and an enormous disclaimer - both of which I had planned had I continued. It's not possible to invoke external programs or call into other languages from Zig comptime[2], that would require a separate build step. One idea I considered is to have the parser recognize the comments generated by `capnp -o capnp` and use them for validation, so that it's at least easier to get that extra assurance. Loading a file in schema.capnp format could also be done, of course, but then one might as well introduce a separate build step. 1) https://code.blicky.net/yorhel/capnzig/src/commit/5618819e01c8dda38962b986e4107d7d20e863f5/src/schema.zig#L812 2) Okay, Zig's comptime being turing complete one could potentially write a C++ interpreter and run capnp's layouting code during Zig compilation, but let's not go there. :) Yorhel. On 2023-01-30, 'Kenton Varda' via Cap'n Proto wrote: > Hi Yorhel, > > Cool stuff. It's neat that Zig can run code at compile time. > > Question, though: are you calling the C++ parser library to parse Cap'n > Proto schema language, or did you write a whole new parser for it? If the > latter, I'm worried as the algorithm to decide the layout of structs is > quite complex, especially when unions of groups are involved. If you are > re-implementing that, it would be pretty easy to get wrong leading to > subtle incompatibilities. Is it possible to invoke the parser from the C++ > implementation as a library, to convert from capnp IDL into schema.capnp > format, which could then be interpreted by your implementation? > > -Kenton > > On Thu, Jan 12, 2023 at 6:06 AM 'Yorhel' via Cap'n Proto < > capnproto@googlegroups.com> wrote: > > > Hi list, > > > > I've been working on a Cap'n Proto implementation for the Zig language. > > This implementation is unusual in that it leverages the language's > > compile-time code execution capabilities to read a schema file - written > > in Cap'n Proto's schema language, no separate compilation step required > > - and then make the interfaces defined in the schema available to > > library users. > > > > I wasn't 100% sure if this was going to work, but I now have a parser, > > struct layouting algorithm and some rudimentary experiments with a > > type-safe struct read/write API that compiles down to code that *should* > > be just as efficient as if a code generation step was involved. > > > > Sadly, now that I've convinced myself that this might actually work out, > > I've lost the motivation to do the remaining 80% of the work and turn it > > into a usable library. So instead I'm publishing the unfinished product > > with the hope that it might inspire or prove useful to someone. And, who > > knows, maybe I'll continue to work on it some time in the future. > > > > The code is at https://code.blicky.net/yorhel/capnzig > > > > Documentation's kind of lacking - the code is commented here and there, > > but making sense of everything may take effort. I'll be around to answer > > questions if there are any. > > > > Here's hoping we'll have a mature Cap'n Proto implementation for Zig at > > some point, regardless of which code generation strategy is taken. > > > > Yorhel. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Cap'n Proto" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to capnproto+unsubscr...@googlegroups.com. > > To view this discussion on the web visit > > https://groups.google.com/d/msgid/capnproto/Y7/3v1VZfkFIFO5s%40gmai021. > > > > -- > You received this message because you are subscribed to the Google Groups > "Cap'n Proto" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to capnproto+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DFvmUpqCvqEtGAvU4gtMo5V8drrGcPHXWRuPBYc254PA%40mail.gmail.com. -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/Y9jOyhsNXGK3areQ%40gmai021.
Re: [capnproto] Incomplete compile-time Zig implementation
(Sorry for the very slow reply, my inbox is a mess.) -Kenton On Mon, Jan 30, 2023 at 6:09 PM Kenton Varda wrote: > Hi Yorhel, > > Cool stuff. It's neat that Zig can run code at compile time. > > Question, though: are you calling the C++ parser library to parse Cap'n > Proto schema language, or did you write a whole new parser for it? If the > latter, I'm worried as the algorithm to decide the layout of structs is > quite complex, especially when unions of groups are involved. If you are > re-implementing that, it would be pretty easy to get wrong leading to > subtle incompatibilities. Is it possible to invoke the parser from the C++ > implementation as a library, to convert from capnp IDL into schema.capnp > format, which could then be interpreted by your implementation? > > -Kenton > > On Thu, Jan 12, 2023 at 6:06 AM 'Yorhel' via Cap'n Proto < > capnproto@googlegroups.com> wrote: > >> Hi list, >> >> I've been working on a Cap'n Proto implementation for the Zig language. >> This implementation is unusual in that it leverages the language's >> compile-time code execution capabilities to read a schema file - written >> in Cap'n Proto's schema language, no separate compilation step required >> - and then make the interfaces defined in the schema available to >> library users. >> >> I wasn't 100% sure if this was going to work, but I now have a parser, >> struct layouting algorithm and some rudimentary experiments with a >> type-safe struct read/write API that compiles down to code that *should* >> be just as efficient as if a code generation step was involved. >> >> Sadly, now that I've convinced myself that this might actually work out, >> I've lost the motivation to do the remaining 80% of the work and turn it >> into a usable library. So instead I'm publishing the unfinished product >> with the hope that it might inspire or prove useful to someone. And, who >> knows, maybe I'll continue to work on it some time in the future. >> >> The code is at https://code.blicky.net/yorhel/capnzig >> >> Documentation's kind of lacking - the code is commented here and there, >> but making sense of everything may take effort. I'll be around to answer >> questions if there are any. >> >> Here's hoping we'll have a mature Cap'n Proto implementation for Zig at >> some point, regardless of which code generation strategy is taken. >> >> Yorhel. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Cap'n Proto" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to capnproto+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/capnproto/Y7/3v1VZfkFIFO5s%40gmai021. >> > -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/CAJouXQnHZsYcGMMtQW_Gu74uZu%2BnALdCRkL_XPZiP7x63UD-Lg%40mail.gmail.com.
Re: [capnproto] Incomplete compile-time Zig implementation
Hi Yorhel, Cool stuff. It's neat that Zig can run code at compile time. Question, though: are you calling the C++ parser library to parse Cap'n Proto schema language, or did you write a whole new parser for it? If the latter, I'm worried as the algorithm to decide the layout of structs is quite complex, especially when unions of groups are involved. If you are re-implementing that, it would be pretty easy to get wrong leading to subtle incompatibilities. Is it possible to invoke the parser from the C++ implementation as a library, to convert from capnp IDL into schema.capnp format, which could then be interpreted by your implementation? -Kenton On Thu, Jan 12, 2023 at 6:06 AM 'Yorhel' via Cap'n Proto < capnproto@googlegroups.com> wrote: > Hi list, > > I've been working on a Cap'n Proto implementation for the Zig language. > This implementation is unusual in that it leverages the language's > compile-time code execution capabilities to read a schema file - written > in Cap'n Proto's schema language, no separate compilation step required > - and then make the interfaces defined in the schema available to > library users. > > I wasn't 100% sure if this was going to work, but I now have a parser, > struct layouting algorithm and some rudimentary experiments with a > type-safe struct read/write API that compiles down to code that *should* > be just as efficient as if a code generation step was involved. > > Sadly, now that I've convinced myself that this might actually work out, > I've lost the motivation to do the remaining 80% of the work and turn it > into a usable library. So instead I'm publishing the unfinished product > with the hope that it might inspire or prove useful to someone. And, who > knows, maybe I'll continue to work on it some time in the future. > > The code is at https://code.blicky.net/yorhel/capnzig > > Documentation's kind of lacking - the code is commented here and there, > but making sense of everything may take effort. I'll be around to answer > questions if there are any. > > Here's hoping we'll have a mature Cap'n Proto implementation for Zig at > some point, regardless of which code generation strategy is taken. > > Yorhel. > > -- > You received this message because you are subscribed to the Google Groups > "Cap'n Proto" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to capnproto+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/capnproto/Y7/3v1VZfkFIFO5s%40gmai021. > -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DFvmUpqCvqEtGAvU4gtMo5V8drrGcPHXWRuPBYc254PA%40mail.gmail.com.
[capnproto] Incomplete compile-time Zig implementation
Hi list, I've been working on a Cap'n Proto implementation for the Zig language. This implementation is unusual in that it leverages the language's compile-time code execution capabilities to read a schema file - written in Cap'n Proto's schema language, no separate compilation step required - and then make the interfaces defined in the schema available to library users. I wasn't 100% sure if this was going to work, but I now have a parser, struct layouting algorithm and some rudimentary experiments with a type-safe struct read/write API that compiles down to code that *should* be just as efficient as if a code generation step was involved. Sadly, now that I've convinced myself that this might actually work out, I've lost the motivation to do the remaining 80% of the work and turn it into a usable library. So instead I'm publishing the unfinished product with the hope that it might inspire or prove useful to someone. And, who knows, maybe I'll continue to work on it some time in the future. The code is at https://code.blicky.net/yorhel/capnzig Documentation's kind of lacking - the code is commented here and there, but making sense of everything may take effort. I'll be around to answer questions if there are any. Here's hoping we'll have a mature Cap'n Proto implementation for Zig at some point, regardless of which code generation strategy is taken. Yorhel. -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/Y7/3v1VZfkFIFO5s%40gmai021.
