On Mon, Sep 19, 2016 at 12:30:53PM +0300, Nadav Har'El wrote:
> On Mon, Sep 19, 2016 at 11:55 AM, Avi Kivity <a...@scylladb.com> wrote:
> >
> > On 09/19/2016 11:49 AM, Nadav Har'El wrote:
> >
> >
> > On Mon, Sep 19, 2016 at 10:52 AM, Gleb Natapov <g...@scylladb.com> wrote:
> >
> >>
> >> > 2. I'm afraid the scheduler thing might only be the tip of the iceberg
> >> of
> >> > problems caused by this lazy TLB thing. Could we have, for example (and
> >> > this is just a hypothetical example) one thread doing a write() to disk
> >> of
> >> > some data from an mmap'ed area, and this data is supposed to be read by
> >> a
> >> > ZFS thread which runs on a different CPU - and because it is labeled a
> >> > "system thread", it won't do a TLB flush before reading the mmap'ed
> >> area?
> >> write() copies data into ZFS ARC.
> >>
> >
> >> > Why are we confident that "system threads" never need to read user's
> >> > mmap'ed data?
> >> >
> >> If they do this is a bug as you discovered. They may access mmaped
> >> memory, but they should do so through their own mappings.
> >
> >
> > So basically, "system threads" (and the scheduler) are not allowed to read
> > any user memory, because any user memory may be mmap'ed.
> > Whatever user memory is needed in a system thread, must first be *copied*
> > in the original user thread...
> >
> >
> > Or, the mapping must be pinned (and paged in) for the duration of the
> > access.
> >
> So if I understand correctly, both your and Gleb's sentiment is there is no
> need not to make the scheduler more forgiving for page table changes, but
> rather require that sched::threads objects live in a mapping which could
> never change - which more-or-less means in the OSv case, that sched::thread
> objects should be allocated only on the heap.
As most other things this is trade-off. Allowing sched::threads to live
in a stack is not worth removing the optimization for, but may be something
else will provide so much benefit that allowing non app threads accessing
mmaped area will be worth supporting. Optimization may still avoid
sending IPI to idle cpus.

> So a patch which will enforce sched::thread to be allocated only on the
> heap is the right way to go here? (I sent such a patch already, but I'll
> try to send a slightly less ugly one. In any case it will be a long patch).
I think so. BTW can this be fixed by forcing tlb flush when sending new
thread to another cpu somehow?

> By the way, it is obvious that a sched::thread cannot live in a paged-out
> mapping, as the scheduler cannot sleep to wait for it. But the issue here
> is more subtle: stack memory is guaranteed to be always paged in (for
> reasons explained here https://github.com/cloudius-systems/osv/issues/143)
> - and nontheless, it is still not fine to put a sched::thread there because
> of the TLB flushing issue.


You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to