Hmm when I say "async I/O" I mean the same thing as "non-blocking I/O".
I haven't worked in C#, but have done some stuff with the Win32 C API, where they call it "overlapped" I/O. -Kenton On Wed, Mar 3, 2021 at 3:54 PM John Demme <[email protected]> wrote: > I think in Windows, async IO is different from non-blocking IO. I haven't > used Windows async IO, other than briefly trying to use C# async IO, > getting very confused, and (perhaps prematurely) that C# async is intended > to be async everywhere (without some really nasty hacks) and you can only > use async IO in async code. > > ~John > > On Wed, Mar 3, 2021 at 1:42 PM Kenton Varda <[email protected]> wrote: > >> On Wed, Mar 3, 2021 at 2:11 PM John Demme <[email protected]> wrote: >> >>> Some operating systems (*cough* Windows *cough*) don't support >>> non-blocking file (or inter-process pipe) I/O without some unreliable >>> hacks. AFAICT. >>> >> >> Hate to say it, but my understanding is that Windows has much more robust >> async filesystem support than Linux does. The comment that Pepijn found is >> about Linux, not Windows. >> >> Linux filesystem implementations are not async (at least for metadata / >> directory walking), therefore the only way to access them in an async way >> is to use threads -- either in the kernel, or in userspace. The kernel >> isn't very excited about spawning kernel threads transparently -- it would >> rather than you create userspace threads and make blocking calls. >> >> That said, this is changing a bit with io_uring, where kernel threads >> running asynchronously from userspace threads are becoming more of a thing. >> Still, on the kernel side, there are threads, even if it looks like "async" >> I/O from the userspace side. >> >> Disclaimer: I'm not a kernel developer, this is my second-hand >> understanding. Also, I know almost nothing about Windows, except that async >> file I/O has featured much more prominently in the Win32 API going back >> decades, so I'm guessing it's also baked deeper into the kernel. >> >> -Kenton >> >> >>> >>> ~John >>> >>> On Wed, Mar 3, 2021 at 11:02 AM pepijn de vos <[email protected]> >>> wrote: >>> >>>> Oh I just went hunting for the asynch bits and in the header it >>>> actually say there is no such thing as async file IO >>>> >>>> // >>>> ======================================================================================= >>>> // The filesystem API >>>> // >>>> // This API is strictly synchronous because, unfortunately, there's no >>>> such thing as asynchronous >>>> // filesystem access in practice. The filesystem drivers on Linux are >>>> written to assume they can >>>> // block. The AIO API is only actually asynchronous for >>>> reading/writing the raw file blocks, but if >>>> // the filesystem needs to be involved (to allocate blocks, update >>>> metadata, etc.) that will block. >>>> // It's best to imagine that the filesystem is just another tier of >>>> memory that happens to be >>>> // slower than RAM (which is slower than L3 cache, which is slower >>>> than L2, which is slower than >>>> // L1). You can't do asynchronous RAM access so why asynchronous >>>> filesystem? The only way to >>>> // parallelize these is using threads. >>>> // >>>> // All KJ filesystem objects are thread-safe, and so all methods are >>>> marked "const" (even write >>>> // methods). Of course, if you concurrently write the same bytes of a >>>> file from multiple threads, >>>> // it's unspecified which write will "win". >>>> >>>> On Wed, Mar 3, 2021 at 7:57 PM pepijn de vos <[email protected]> >>>> wrote: >>>> >>>>> Hey Kenton, >>>>> >>>>> Thanks for the answer, and sorry my frustration in trying to find out >>>>> how to do async file IO in cap'n proto came across rude. >>>>> >>>>> What puzzles me is your suggestion that I don't need to use KJ. >>>>> What puzzles me even more is that the KJ file I just obtained seems to >>>>> not support async operations at all. >>>>> Are you suggesting it's actually fine to just do blocking IO in the >>>>> RPC eventloop? >>>>> Is there some KJ thing to do async file IO, or a way to use other >>>>> libraries for async file IO with the KJ eventloop? >>>>> >>>>> Thanks to all the people working on cap'n proto, it seems pretty neat >>>>> :) >>>>> >>>>> Cheers, >>>>> Pepijn >>>>> >>>>> On Wed, Mar 3, 2021 at 4:42 PM Kenton Varda <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Pepijn, >>>>>> >>>>>> The full comment you refer to says: >>>>>> >>>>>> // DO NOT CALL THIS except at the top level of your program, e.g. in >>>>>> main(). Anywhere else, you >>>>>> // should instead have your caller pass in a Filesystem object, or a >>>>>> specific Directory object, >>>>>> // or whatever it is that your code needs. This ensures that your >>>>>> code supports dependency >>>>>> // injection, which makes it more reusable and testable. >>>>>> >>>>>> As the comment says, the idea is that you construct your Filesystem >>>>>> object at the top level of your program, then you pass it (or specific >>>>>> File >>>>>> or Directory objects) down to whatever parts of your program need it. >>>>>> That >>>>>> ensures that those parts of your program can easily be unit-tested >>>>>> against >>>>>> a fake in-memory filesystem, by swapping out the objects that you pass >>>>>> down. This is trying to help you make your code more flexible, but you >>>>>> can >>>>>> ignore it if you want. >>>>>> >>>>>> You also don't have to use the KJ filesystem API. In fact, nothing >>>>>> else in KJ or Cap'n Proto cares what you use to open files. KJ is a >>>>>> toolkit, meaning it provides a bunch of tools, but doesn't force you to >>>>>> use >>>>>> them if you don't want to. >>>>>> >>>>>> KJ documentation can be found here: >>>>>> https://github.com/capnproto/capnproto/tree/master/kjdoc >>>>>> >>>>>> Please note that Cap'n Proto and KJ are open source software provided >>>>>> for free. I'd like them to be useful but I don't get any actual benefit >>>>>> from you using them, and I'm not interested in helping people who are >>>>>> rude. >>>>>> So, if you have further questions, please try to tone it down a bit. >>>>>> Thanks. >>>>>> >>>>>> -Kenton >>>>>> >>>>>> On Wed, Mar 3, 2021 at 5:54 AM [email protected] < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hey, >>>>>>> >>>>>>> I'm just getting started and pretty confused. >>>>>>> The RPC server is all async and stuff, so no blocking IO in there. >>>>>>> Then there are some vague mentions that you should use KJ for most >>>>>>> things. >>>>>>> But KJ documentation is nowhere to be found... >>>>>>> >>>>>>> All I want to do is just open a file and write to it. >>>>>>> After resorting to reading the source code, I found >>>>>>> >>>>>>> https://github.com/capnproto/capnproto/blob/master/c%2B%2B/src/kj/filesystem.h >>>>>>> So I managed to create a path so far. >>>>>>> Then I started to look for how to open a file. >>>>>>> I found some methods on Directory. >>>>>>> So then I started to look how to make a directory. >>>>>>> Then I found a Filesystem, which is abstract. >>>>>>> Then I found newDiskFilesystem which tells me >>>>>>> DO NOT CALL THIS except in main() >>>>>>> Well **** how am I supposed to open a file inside my server handler >>>>>>> then?? >>>>>>> >>>>>>> Pepijn >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Cap'n Proto" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/capnproto/cd5ce66c-b12e-4612-b383-32462f047f69n%40googlegroups.com >>>>>>> <https://groups.google.com/d/msgid/capnproto/cd5ce66c-b12e-4612-b383-32462f047f69n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Cap'n Proto" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/capnproto/CANPfQguo2bq4G0tWPeAgzphdSpU4isMkxRUzNmxjCjyvvaxO0w%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/capnproto/CANPfQguo2bq4G0tWPeAgzphdSpU4isMkxRUzNmxjCjyvvaxO0w%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/CAJouXQnKB7_Epb5JZ--L%2B%3DNbGKn8prSN1Yyi%3Dc_L3Hm%2BSdirvw%40mail.gmail.com.
