I could easily be confusing the C# API with the Win32 API, but I recall seeing some StackOverflow posts which indicated there was no way to do what I wanted to do. This was some time ago and my memory ain't so great. My hack-y solution was to spin up a thread for each file/pipe I needed to read from and send all the data down a thread safe queue, which could then implement a 'select()' like mechanism, which is what I really wanted.
... but I no longer have to care about that. At least until I do again. ~John On Wed, Mar 3, 2021 at 2:02 PM Kenton Varda <[email protected]> wrote: > Hmm when I say "async I/O" I mean the same thing as "non-blocking I/O". > > I haven't worked in C#, but have done some stuff with the Win32 C API, > where they call it "overlapped" I/O. > > -Kenton > > On Wed, Mar 3, 2021 at 3:54 PM John Demme <[email protected]> wrote: > >> I think in Windows, async IO is different from non-blocking IO. I haven't >> used Windows async IO, other than briefly trying to use C# async IO, >> getting very confused, and (perhaps prematurely) that C# async is intended >> to be async everywhere (without some really nasty hacks) and you can only >> use async IO in async code. >> >> ~John >> >> On Wed, Mar 3, 2021 at 1:42 PM Kenton Varda <[email protected]> >> wrote: >> >>> On Wed, Mar 3, 2021 at 2:11 PM John Demme <[email protected]> wrote: >>> >>>> Some operating systems (*cough* Windows *cough*) don't support >>>> non-blocking file (or inter-process pipe) I/O without some unreliable >>>> hacks. AFAICT. >>>> >>> >>> Hate to say it, but my understanding is that Windows has much more >>> robust async filesystem support than Linux does. The comment that Pepijn >>> found is about Linux, not Windows. >>> >>> Linux filesystem implementations are not async (at least for metadata / >>> directory walking), therefore the only way to access them in an async way >>> is to use threads -- either in the kernel, or in userspace. The kernel >>> isn't very excited about spawning kernel threads transparently -- it would >>> rather than you create userspace threads and make blocking calls. >>> >>> That said, this is changing a bit with io_uring, where kernel threads >>> running asynchronously from userspace threads are becoming more of a thing. >>> Still, on the kernel side, there are threads, even if it looks like "async" >>> I/O from the userspace side. >>> >>> Disclaimer: I'm not a kernel developer, this is my second-hand >>> understanding. Also, I know almost nothing about Windows, except that async >>> file I/O has featured much more prominently in the Win32 API going back >>> decades, so I'm guessing it's also baked deeper into the kernel. >>> >>> -Kenton >>> >>> >>>> >>>> ~John >>>> >>>> On Wed, Mar 3, 2021 at 11:02 AM pepijn de vos <[email protected]> >>>> wrote: >>>> >>>>> Oh I just went hunting for the asynch bits and in the header it >>>>> actually say there is no such thing as async file IO >>>>> >>>>> // >>>>> ======================================================================================= >>>>> // The filesystem API >>>>> // >>>>> // This API is strictly synchronous because, unfortunately, there's >>>>> no such thing as asynchronous >>>>> // filesystem access in practice. The filesystem drivers on Linux are >>>>> written to assume they can >>>>> // block. The AIO API is only actually asynchronous for >>>>> reading/writing the raw file blocks, but if >>>>> // the filesystem needs to be involved (to allocate blocks, update >>>>> metadata, etc.) that will block. >>>>> // It's best to imagine that the filesystem is just another tier of >>>>> memory that happens to be >>>>> // slower than RAM (which is slower than L3 cache, which is slower >>>>> than L2, which is slower than >>>>> // L1). You can't do asynchronous RAM access so why asynchronous >>>>> filesystem? The only way to >>>>> // parallelize these is using threads. >>>>> // >>>>> // All KJ filesystem objects are thread-safe, and so all methods are >>>>> marked "const" (even write >>>>> // methods). Of course, if you concurrently write the same bytes of a >>>>> file from multiple threads, >>>>> // it's unspecified which write will "win". >>>>> >>>>> On Wed, Mar 3, 2021 at 7:57 PM pepijn de vos <[email protected]> >>>>> wrote: >>>>> >>>>>> Hey Kenton, >>>>>> >>>>>> Thanks for the answer, and sorry my frustration in trying to find out >>>>>> how to do async file IO in cap'n proto came across rude. >>>>>> >>>>>> What puzzles me is your suggestion that I don't need to use KJ. >>>>>> What puzzles me even more is that the KJ file I just obtained seems >>>>>> to not support async operations at all. >>>>>> Are you suggesting it's actually fine to just do blocking IO in the >>>>>> RPC eventloop? >>>>>> Is there some KJ thing to do async file IO, or a way to use other >>>>>> libraries for async file IO with the KJ eventloop? >>>>>> >>>>>> Thanks to all the people working on cap'n proto, it seems pretty neat >>>>>> :) >>>>>> >>>>>> Cheers, >>>>>> Pepijn >>>>>> >>>>>> On Wed, Mar 3, 2021 at 4:42 PM Kenton Varda <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Pepijn, >>>>>>> >>>>>>> The full comment you refer to says: >>>>>>> >>>>>>> // DO NOT CALL THIS except at the top level of your program, e.g. in >>>>>>> main(). Anywhere else, you >>>>>>> // should instead have your caller pass in a Filesystem object, or a >>>>>>> specific Directory object, >>>>>>> // or whatever it is that your code needs. This ensures that your >>>>>>> code supports dependency >>>>>>> // injection, which makes it more reusable and testable. >>>>>>> >>>>>>> As the comment says, the idea is that you construct your Filesystem >>>>>>> object at the top level of your program, then you pass it (or specific >>>>>>> File >>>>>>> or Directory objects) down to whatever parts of your program need it. >>>>>>> That >>>>>>> ensures that those parts of your program can easily be unit-tested >>>>>>> against >>>>>>> a fake in-memory filesystem, by swapping out the objects that you pass >>>>>>> down. This is trying to help you make your code more flexible, but you >>>>>>> can >>>>>>> ignore it if you want. >>>>>>> >>>>>>> You also don't have to use the KJ filesystem API. In fact, nothing >>>>>>> else in KJ or Cap'n Proto cares what you use to open files. KJ is a >>>>>>> toolkit, meaning it provides a bunch of tools, but doesn't force you to >>>>>>> use >>>>>>> them if you don't want to. >>>>>>> >>>>>>> KJ documentation can be found here: >>>>>>> https://github.com/capnproto/capnproto/tree/master/kjdoc >>>>>>> >>>>>>> Please note that Cap'n Proto and KJ are open source software >>>>>>> provided for free. I'd like them to be useful but I don't get any actual >>>>>>> benefit from you using them, and I'm not interested in helping people >>>>>>> who >>>>>>> are rude. So, if you have further questions, please try to tone it down >>>>>>> a >>>>>>> bit. Thanks. >>>>>>> >>>>>>> -Kenton >>>>>>> >>>>>>> On Wed, Mar 3, 2021 at 5:54 AM [email protected] < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hey, >>>>>>>> >>>>>>>> I'm just getting started and pretty confused. >>>>>>>> The RPC server is all async and stuff, so no blocking IO in there. >>>>>>>> Then there are some vague mentions that you should use KJ for most >>>>>>>> things. >>>>>>>> But KJ documentation is nowhere to be found... >>>>>>>> >>>>>>>> All I want to do is just open a file and write to it. >>>>>>>> After resorting to reading the source code, I found >>>>>>>> >>>>>>>> https://github.com/capnproto/capnproto/blob/master/c%2B%2B/src/kj/filesystem.h >>>>>>>> So I managed to create a path so far. >>>>>>>> Then I started to look for how to open a file. >>>>>>>> I found some methods on Directory. >>>>>>>> So then I started to look how to make a directory. >>>>>>>> Then I found a Filesystem, which is abstract. >>>>>>>> Then I found newDiskFilesystem which tells me >>>>>>>> DO NOT CALL THIS except in main() >>>>>>>> Well **** how am I supposed to open a file inside my server handler >>>>>>>> then?? >>>>>>>> >>>>>>>> Pepijn >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Cap'n Proto" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/capnproto/cd5ce66c-b12e-4612-b383-32462f047f69n%40googlegroups.com >>>>>>>> <https://groups.google.com/d/msgid/capnproto/cd5ce66c-b12e-4612-b383-32462f047f69n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Cap'n Proto" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/capnproto/CANPfQguo2bq4G0tWPeAgzphdSpU4isMkxRUzNmxjCjyvvaxO0w%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/capnproto/CANPfQguo2bq4G0tWPeAgzphdSpU4isMkxRUzNmxjCjyvvaxO0w%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/CAOjmg%3DvY10X8N7SjXUj6%3DFJRE0PGw%2Bi%3DmDfkCdHYpJtb1iuNWQ%40mail.gmail.com.
