On 10/07/2015 10:52 PM, Sage Weil wrote: > On Wed, 7 Oct 2015, David Zafman wrote: >> There would be a benefit to doing fadvise POSIX_FADV_DONTNEED after >> deep-scrub reads for objects not recently accessed by clients. > Yeah, it's the 'except for stuff already in cache' part that we don't do > (and the kernel doesn't give us a good interface for). IIRC there was a > patch that guessed based on whether the obc was already in cache, which > seems like a pretty decent heuristic, but I forget if that was in the > final version.
I've run some tests and it look like on XFS cache is discarded on O_DIRECT write and read but on EXT4 is discarded only on O_DIRECT write. I've found some patches to add support for "read only if in page cache" (preadv2/RWF_NONBLOCK) but can't find them in kernel source. Maybe Milosz Tanski can tell more about that. I think it could help a bit during deep scrub. >> I see the NewStore objectstore sometimes using the O_DIRECT flag for writes. >> This concerns me because the open(2) man pages says: >> >> "Applications should avoid mixing O_DIRECT and normal I/O to the same file, >> and especially to overlapping byte regions in the same file. Even when the >> filesystem correctly handles the coherency issues in this situation, overall >> I/O throughput is likely to be slower than using either mode alone." > Yeah: an O_DIRECT write will do a cache flush on the write range, so if > there was already dirty data in cache you'll write twice. There's > similarly an invalidate on read. I need to go back through the newstore > code and see how the modes are being mixed and how it can be avoided... > > sage > > >> On 10/7/15 7:50 AM, Sage Weil wrote: >>> It's not, but it would not be ahrd to do this. There are fadvise-style >>> hints being passed down that could trigger O_DIRECT reads in this case. >>> That may not be the best choice, though--it won't use data that happens >>> to be in cache and it'll also throw it out.. >>> >>> On Wed, 7 Oct 2015, Pawe? Sadowski wrote: >>> >>>> Hi, >>>> >>>> Can anyone tell if deep scrub is done using O_DIRECT flag or not? I'm >>>> not able to verify that in source code. >>>> >>>> If not would it be possible to add such feature (maybe config option) to >>>> help keeping Linux page cache in better shape? >>>> >>>> Thanks, -- PS _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
