On Mon, Feb 4, 2019 at 10:10 PM Gregory Farnum <gfar...@redhat.com> wrote:
> > On Mon, Feb 4, 2019 at 8:03 AM Mahmoud Ismail < > mahmoudahmedism...@gmail.com> wrote: > >> On Mon, Feb 4, 2019 at 4:35 PM Gregory Farnum <gfar...@redhat.com> wrote: >> >>> >>> >>> On Mon, Feb 4, 2019 at 7:32 AM Mahmoud Ismail < >>> mahmoudahmedism...@gmail.com> wrote: >>> >>>> On Mon, Feb 4, 2019 at 4:16 PM Gregory Farnum <gfar...@redhat.com> >>>> wrote: >>>> >>>>> On Fri, Feb 1, 2019 at 2:29 AM Mahmoud Ismail < >>>>> mahmoudahmedism...@gmail.com> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> I'm a bit confused about how the journaling actually works in the >>>>>> MDS. >>>>>> >>>>>> I was reading about these two configuration parameters (journal write >>>>>> head interval) and (mds early reply). Does the MDS flush the journal >>>>>> synchronously after each operation? and by setting mds eary reply to true >>>>>> it allows operations to return without flushing. If so, what the other >>>>>> parameter (journal write head interval) do or isn't it for MDS?. Also, >>>>>> can >>>>>> all operations return without flushing with the mds early reply or is it >>>>>> specific to a subset of operations?. >>>>>> >>>>> >>>>> In general, the MDS journal is flushed every five seconds (by >>>>> default), and client requests get an early reply when the operation is >>>>> done >>>>> in memory but not yet committed to RADOS. Some operations will trigger an >>>>> immediate flush, and there may be some operations that can't get an early >>>>> reply or that need to wait for part of the operation to get committed >>>>> (like >>>>> renames that move a file's authority to a different MDS). >>>>> IIRC the journal write head interval controls how often it flushes out >>>>> the journal's header, which limits how out-of-date its hints on restart >>>>> can >>>>> be. (When the MDS restarts, it asks the journal head where the journal's >>>>> unfinished start and end points are, but of course more of the journaled >>>>> operations may have been fully completed since the head was written.) >>>>> >>>> >>>> Thanks for the explanation. Which operations trigger an immediate >>>> flush? Is the readdir one of these operations?. I noticed that the readdir >>>> operation latency is going higher under load when the OSDs are hitting the >>>> limit of the underlying hdd throughput. Can i assume that this is happening >>>> due to the journal flushing then? >>>> >>> >>> Not directly, but a readdir might ask to know the size of each file and >>> that will force the other clients in the system to flush their dirty data >>> in the directory (so that the readdir can return valid results). >>> -Greg >>> >>> >> >> Could it be also due to the MDS lock (operations waiting for the lock >> under load)? >> > > Well that's not going to cause high OSD usage, and the MDS lock is not > held while writes are happening. But if the MDS is using 100% CPU, yes, it > could be contended. > > Yes, i guess that will be the case if i'm writing actual data to the OSDs, however, i was testing only the metadata operations on the MDSs. So under high load of metadata operations, the lock contention will be more apparent. > Also, i assume that the journal is using a different thread for flushing, >> Right? >> > > Yes, that's correct. > >> >>>>>>
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
