niyue commented on pull request #11486: URL: https://github.com/apache/arrow/pull/11486#issuecomment-954340112
@lidavidm > we will want a unit test to ensure the bytes read is as expected Sure. I will look into it how more unit tests can be added. > Additionally, another candidate for a follow-up item is to include the I/O coalescer so that we don't suffer on remote filesystems. In my test under Linux, I found Linux will do read ahead IO. In my limited testing, depending on read ahead configuration in Linux, the IO may be 2x than the minimum necessary if the access pattern is random access and the persisted record batch is small. I don't look into how `S3FileSystem` handles this, but even on local file system, `posix_fadvise` is desirable to advise operating system the access pattern. Currently, `file.cc` has some support for `POSIX_FADV_WILLNEED`, it will be great if other patterns can be supported there, but this is likely another independent area to improve. > I think IoRecordedRandomAccessFile should be moved into reader.cc No problem. I will move it into reader.cc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
