I think in Windows, async IO is different from non-blocking IO. I haven't used Windows async IO, other than briefly trying to use C# async IO, getting very confused, and (perhaps prematurely) that C# async is intended to be async everywhere (without some really nasty hacks) and you can only use async IO in async code.
~John On Wed, Mar 3, 2021 at 1:42 PM Kenton Varda <[email protected]> wrote: > On Wed, Mar 3, 2021 at 2:11 PM John Demme <[email protected]> wrote: > >> Some operating systems (*cough* Windows *cough*) don't support >> non-blocking file (or inter-process pipe) I/O without some unreliable >> hacks. AFAICT. >> > > Hate to say it, but my understanding is that Windows has much more robust > async filesystem support than Linux does. The comment that Pepijn found is > about Linux, not Windows. > > Linux filesystem implementations are not async (at least for metadata / > directory walking), therefore the only way to access them in an async way > is to use threads -- either in the kernel, or in userspace. The kernel > isn't very excited about spawning kernel threads transparently -- it would > rather than you create userspace threads and make blocking calls. > > That said, this is changing a bit with io_uring, where kernel threads > running asynchronously from userspace threads are becoming more of a thing. > Still, on the kernel side, there are threads, even if it looks like "async" > I/O from the userspace side. > > Disclaimer: I'm not a kernel developer, this is my second-hand > understanding. Also, I know almost nothing about Windows, except that async > file I/O has featured much more prominently in the Win32 API going back > decades, so I'm guessing it's also baked deeper into the kernel. > > -Kenton > > >> >> ~John >> >> On Wed, Mar 3, 2021 at 11:02 AM pepijn de vos <[email protected]> >> wrote: >> >>> Oh I just went hunting for the asynch bits and in the header it actually >>> say there is no such thing as async file IO >>> >>> // >>> ======================================================================================= >>> // The filesystem API >>> // >>> // This API is strictly synchronous because, unfortunately, there's no >>> such thing as asynchronous >>> // filesystem access in practice. The filesystem drivers on Linux are >>> written to assume they can >>> // block. The AIO API is only actually asynchronous for reading/writing >>> the raw file blocks, but if >>> // the filesystem needs to be involved (to allocate blocks, update >>> metadata, etc.) that will block. >>> // It's best to imagine that the filesystem is just another tier of >>> memory that happens to be >>> // slower than RAM (which is slower than L3 cache, which is slower than >>> L2, which is slower than >>> // L1). You can't do asynchronous RAM access so why asynchronous >>> filesystem? The only way to >>> // parallelize these is using threads. >>> // >>> // All KJ filesystem objects are thread-safe, and so all methods are >>> marked "const" (even write >>> // methods). Of course, if you concurrently write the same bytes of a >>> file from multiple threads, >>> // it's unspecified which write will "win". >>> >>> On Wed, Mar 3, 2021 at 7:57 PM pepijn de vos <[email protected]> >>> wrote: >>> >>>> Hey Kenton, >>>> >>>> Thanks for the answer, and sorry my frustration in trying to find out >>>> how to do async file IO in cap'n proto came across rude. >>>> >>>> What puzzles me is your suggestion that I don't need to use KJ. >>>> What puzzles me even more is that the KJ file I just obtained seems to >>>> not support async operations at all. >>>> Are you suggesting it's actually fine to just do blocking IO in the RPC >>>> eventloop? >>>> Is there some KJ thing to do async file IO, or a way to use other >>>> libraries for async file IO with the KJ eventloop? >>>> >>>> Thanks to all the people working on cap'n proto, it seems pretty neat :) >>>> >>>> Cheers, >>>> Pepijn >>>> >>>> On Wed, Mar 3, 2021 at 4:42 PM Kenton Varda <[email protected]> >>>> wrote: >>>> >>>>> Hi Pepijn, >>>>> >>>>> The full comment you refer to says: >>>>> >>>>> // DO NOT CALL THIS except at the top level of your program, e.g. in >>>>> main(). Anywhere else, you >>>>> // should instead have your caller pass in a Filesystem object, or a >>>>> specific Directory object, >>>>> // or whatever it is that your code needs. This ensures that your code >>>>> supports dependency >>>>> // injection, which makes it more reusable and testable. >>>>> >>>>> As the comment says, the idea is that you construct your Filesystem >>>>> object at the top level of your program, then you pass it (or specific >>>>> File >>>>> or Directory objects) down to whatever parts of your program need it. That >>>>> ensures that those parts of your program can easily be unit-tested against >>>>> a fake in-memory filesystem, by swapping out the objects that you pass >>>>> down. This is trying to help you make your code more flexible, but you can >>>>> ignore it if you want. >>>>> >>>>> You also don't have to use the KJ filesystem API. In fact, nothing >>>>> else in KJ or Cap'n Proto cares what you use to open files. KJ is a >>>>> toolkit, meaning it provides a bunch of tools, but doesn't force you to >>>>> use >>>>> them if you don't want to. >>>>> >>>>> KJ documentation can be found here: >>>>> https://github.com/capnproto/capnproto/tree/master/kjdoc >>>>> >>>>> Please note that Cap'n Proto and KJ are open source software provided >>>>> for free. I'd like them to be useful but I don't get any actual benefit >>>>> from you using them, and I'm not interested in helping people who are >>>>> rude. >>>>> So, if you have further questions, please try to tone it down a bit. >>>>> Thanks. >>>>> >>>>> -Kenton >>>>> >>>>> On Wed, Mar 3, 2021 at 5:54 AM [email protected] < >>>>> [email protected]> wrote: >>>>> >>>>>> Hey, >>>>>> >>>>>> I'm just getting started and pretty confused. >>>>>> The RPC server is all async and stuff, so no blocking IO in there. >>>>>> Then there are some vague mentions that you should use KJ for most >>>>>> things. >>>>>> But KJ documentation is nowhere to be found... >>>>>> >>>>>> All I want to do is just open a file and write to it. >>>>>> After resorting to reading the source code, I found >>>>>> >>>>>> https://github.com/capnproto/capnproto/blob/master/c%2B%2B/src/kj/filesystem.h >>>>>> So I managed to create a path so far. >>>>>> Then I started to look for how to open a file. >>>>>> I found some methods on Directory. >>>>>> So then I started to look how to make a directory. >>>>>> Then I found a Filesystem, which is abstract. >>>>>> Then I found newDiskFilesystem which tells me >>>>>> DO NOT CALL THIS except in main() >>>>>> Well **** how am I supposed to open a file inside my server handler >>>>>> then?? >>>>>> >>>>>> Pepijn >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Cap'n Proto" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/capnproto/cd5ce66c-b12e-4612-b383-32462f047f69n%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/capnproto/cd5ce66c-b12e-4612-b383-32462f047f69n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Cap'n Proto" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/capnproto/CANPfQguo2bq4G0tWPeAgzphdSpU4isMkxRUzNmxjCjyvvaxO0w%40mail.gmail.com >>> <https://groups.google.com/d/msgid/capnproto/CANPfQguo2bq4G0tWPeAgzphdSpU4isMkxRUzNmxjCjyvvaxO0w%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/CAOjmg%3Du6STAueobBYoW96VMZhCBwVCGyoLQkVn9SjA36e3G_zQ%40mail.gmail.com.
