Hi, 2015-02-28 16:47 GMT+01:00 Tin Tvrtković <[email protected]>: > I think the peculiarities of doing file I/O in asyncio applications aren't > as well documented as they could be.
Well, they are not documented :-) > * https://code.google.com/p/tulip/wiki/ThirdParty#Filesystem - finally, a > clear message: asyncio does not support asynchronous operations on the > filesystem. Even if files are opened with O_NONBLOCK, read and write will > block I know that asyncio doc lacks to explain how regular files should be handled. I started to document it in this wiki page. > * https://groups.google.com/forum/#!topic/python-tulip/MvpkQeetWZA - can't > be done. Other systems use threads. Use a threadpool (somehow). > * http://stackoverflow.com/questions/3908809/the-state-of-linux-async-io - > if you broaden your search I read that most P2P clients use threads. P2P clients are portable and I/O-bound, disk I/O. So they are good examples of how we can implement it. > What I'm proposing is this: > * for the official docs: > * * a note at the beginning making clear asyncio doesn't generally work on > files, and that the terms 'file descriptor' and 'file-like object' are used > in a more specific, UNIX-type of sense. Also a link to a "Develop with > asyncio" chapter which I'm about to mention I would prefer to not put too much information "at the beginning". I prefer to explain it correctly somewhere (ex: last "Dev" chapter) and repeat the information on each method which may be misused with files with a link to the new section. I just did that for threads. For example, I added "This class is not thread safe." to the BaseEventLoop class: https://docs.python.org/dev/library/asyncio-eventloop.html#asyncio.BaseEventLoop Examples of functions which may be misused with files: - add_reader/writer - remove_reader/writer - connect_read/write_pipe - transports/protocols > * for the examples: > * * an example of serving a small file (read into memory first) using the > default run_in_executor > * * an example of serving a large file using the default run_in_executor > * * an example of serving a large file using a separate process pool > executor I don't think that you can write an short example to handle all aspects of a file. Just say that *all* file methods may be called in a thread. I would prefer to have a third-party library to easily run disk I/O in a thread pool. When the library is stable enough and well tested on various platforms, we can discuss integrating it into asyncio. Such library would be a good place to experiment the support of existing APIs: linux AIO and POSIX AIO. I would be interested to have a better doc of the async I/O support of these libraries. For example, I read things like "Not all devices (including TTY) support AIO (typically return -EINVAL)". So which file systems and which kind of files support async I/O? Which platforms? Solaris have aioread() and aiowrite() for example. It would be nice to have an asyncio API supporting different models of file I/O (threads, linux AIO, posix AIO, etc.) Victor
