Ben Marzinski wrote:

>There is a reproducible memory mapping problem with the s390 SuSE linux
>setup we have.  The bug occurs when two processes have private, read-only
>mappings of the same file and both processes page in the same page at
>the same time. The PTE for that page gets incorrectly marked dirty, which
>causes the page to be marked dirty, and the writepage() address space
>operation to be called. Nothing that the processes have done should have
>caused the page to be written back to the file. The file is modified even
>if the whole filesystem is mounted Read-Only.

There does indeed appear to be a race condition in the s390-specific
memory management backend that could explain the symptom you're seeing.

The problem is that the S/390 does not have a dirty bit that resides
in the PTE like most platforms, and which gets set whenever the processor
writes to the page via the mapping defined by this PTE.  Instead we have a
dirty bit in the 'storage key'; there is one storage key associated with
every physical page.  The dirty bit there is set by the memory subsystem
on every write to the page, no matter via which PTE, and even on DMA
writes by hardware devices.

This means that pages from memory-mapped files would usually start out
with the dirty bit set (because the page was read in from backing store,
and doing so would access the physical page and set the dirty bit).
So we have to reset the dirty bit, which we do when the page is first
mapped into a user address space.  (Note we must only do so on the *first*
mapping to user space, subsequent mapping of a page must not reset a
dirty bit that might have been set in the mean time.)

Unfortunately, this logic would appear to exhibit a race condition
in the situation you're describing.  When a process faults in a page
from a memory-mapped file, the following steps happen in sequence:

- the page is looked up in the page cache; if not found, a new page is
  allocated and added to the page cache.  In any case, at this point
  the page reference count is incremented.

- if the page is not uptodate, a read-in from backing store is triggered
  and the process sleeps until the read has completed

- finally, a page table entry referring to the page is placed into the
  process' page tables.  At this point, our platform-specific hook is
  triggered; we check whether the page count is 1, and if so, reset the
  dirty bit in the storage key.

If the same file is mapped into multiple address spaces, this is broken.
Consider two processes faulting in the page at (nearly) the same time;
both increment the page count, and none of them will see a page count
of 1 when updating the page table, therefore the dirty bit will not be

In later kernels (e.g. SLES-8), this problem is less visible because
we completely ignore the storage key on private read-only mappings as
those can never be dirty anyway.  This was intended as a performance
improvement only (we save storage key operations, which can be expensive),
which is why we didn't backport the new logic.  However, it would have
prevented your symptoms.  (The race is still there, but only on read-
write mappings where it has less obvious effects; some pages may be
written back although there isn't really a need to.)

A proper fix of the race is probably not possible without changes
to common code; a good place to reset the storage key dirty bit might
be SetPageUptodate, but there's no arch-dependent hook there.  We'll
have to think about this ...

>Has anyone else seen this? Does anyone know of any patches to deal with this?
>If anyone wants to see if they can reproduce this, I can send them a copy
>of the program that we wrote to do Step 4 from above. It's less than
>100 lines of C code.

I'd appreciate it if you could send me this program.

Mit freundlichen Gruessen / Best Regards

Ulrich Weigand

  Dr. Ulrich Weigand
  Linux for S/390 Design & Development
  IBM Deutschland Entwicklung GmbH, Schoenaicher Str. 220, 71032 Boeblingen
  Phone: +49-7031/16-3727   ---   Email: [EMAIL PROTECTED]

Reply via email to