Author: akhaldi
Date: Thu Feb 23 12:03:06 2012
New Revision: 55833
URL: http://svn.reactos.org/svn/reactos?rev=55833&view=rev
Log:
[NEWCC]
A reintegration checkpoint for the NewCC branch, brought to you by Team NewCC.
Differences with current ReactOS trunk:
* A new memory area type, MEMORY_AREA_CACHE, is added, which represents a
mapped region of a file. In NEWCC mode, user sections are MEMORY_AREA_CACHE
type as well, and obey new semantics. In non-NEWCC mode, they aren't used.
* A way of claiming a page entry for a specific thread's work is added. Placing
the special SWAPENTRY value MM_WAIT_ENTRY in a page table, or in a section page
table should indicate that memory management code is intended to wait for
another thread to make some status change before checking the state of the page
entry again. In code that uses this convention, a return value of
STATUS_SUCCESS + 1 is used to indicate that the caller should use the
MiWaitForPageEvent macro to wait until somebody has change the state of a wait
entry before checking again. This is a lighter weight mechanism than PAGEOPs.
* A way of asking the caller to perform some blocking operation without locks
held is provided. This replaces some spaghettified code in which locks are
repeatedly taken and broken by code that performs various blocking operations.
Using this mechanism, it is possible to do a small amount of non-blocking work,
fill in a request, then return STATUS_MORE_PROCESSING_REQUIRED to request that
locks be dropped and the blocking operation be carried out. A
MM_REQUIRED_RESOURCES structure is provided to consumers of this contract to
use to accumulate state across many blocking operations. Several functions
wrapping blocking operations are provided in ntoskrnl/cache/reqtools.c.
* Image section pages are no longer direct mapped. This is done to simplify
consolidation of ownership of pages under the data section system. At a later
time, it may be possible to make data pages directly available to image
sections for the same file. This is likely the only direct performance impact
this code makes on non-NEWCC mode.
RMAPs:
* A new type of RMAP entry is introduced, distinguished by
RMAP_IS_SEGMENT(Address) of the rmap entry. This kind of entry contains a
pointer to a section page table node in the Process pointer, which in turn
links back to the MM_SECTION_SEGMENT it belongs to. Therefore, a page belonging
only to a segment (that is, a segment page that isn't mapped) can exist and be
evicted using the normal page eviction mechanism in balance.c. Each of the rmap
function has been modified to deal with segment rmaps.
* The low 8 bits of the Address field in a segment rmap denote the entry number
in the generic table node pointed to by Process that points to the page the
rmap belongs to. By combining them, you can determine the file offset the page
belongs to.
* In NEWCC mode, MmSharePageEntry/UnsharePageEntry are not used, and instead
the page reference count is used to keep track of the number of mappings of a
page, allowing the last reference expiring to allow the page to be recycled
without much intervention. These are still used in non-NEWCC mode. One change
has been made, the count fields have been narrowed by 1 bit to make room for a
dirty bit in SSE entries, needed when a page is present but unmapped.
Section page tables:
* The section page tables are now implemented using RtlGenericTables. This
enables a fairly compact representation of section page tables without having
the existence of a section object imply 4k of fake PDEs. In addition, each node
in the generic table has a wide file offset that is a multiple of 256 pages, or
1 megabyte total. Besides needing wide file offsets, the only other visible
change caused by the switch to generic tables for section page tables is the
need to lock the section segment before interacting with the section page table.
Eviction:
* Page eviction in cache sections is accomplished by MmpPageOutPhysicalAddress.
In the case of a shared page, it tries to remove all mappings of the indicated
page. If this process fails at any point, the page will simply be drawn back
into the target address spaces. After succeeding at this, if TRUE has been
accumulated into the page's dirty bit in the section page table, it is written
back, and then permanently removed.
NewCC mode:
* NEWCC mode is introduced, which rewrites the file cache to a set of cache
stripes actively mapped, along with unmapped section data.
* NewCC is more authentic in its interpretation of the external interface to
the windows cache than the current cache manager, implementing each of the
cache manager functions according to the documented interface with no
preconceived ideas about how anything should be implemented internally. Cache
stripes are implemented on top of section objects, using the same memory
manager paths, and therefore economizing code and complexity. This replaces a
rather complicated system in which pages can be owned by the cache manager and
the memory manager simultaneously and they must cooperate in a fairly
sophisticated way to manage them. Since they're quite interdependent in the
current code, modifying either is very difficult. In NEWCC, they have a clear
division of labor and thus can be worked on independently.
* Several third party filesystems that use the kernel Cc interface work
properly using NEWCC, including matt wu's ext3 driver.
* In contrast with code that tries to make CcInitializeCacheMap and
CcUninitializeCacheMap into a pair that supports reference counting, NEWCC
lazily initializes the shared and private cache maps as needed and uses the
presence of a PrivateCacheMap on at least one file pointing to the
SharedCacheMap as an indication that the FILE_OBJECT reference in the
SharedCacheMap should still be held. When the last PrivateCacheMap is
discarded, that's the appropriate time to tear down caching for a specific
file, as the SharedCacheMap data is allowed to be saved and reused. We honor
this by making the SharedCacheMap into a depot for keeping track of the
PrivateCacheMap objects associated with views of a file.
Modified:
trunk/reactos/ntoskrnl/CMakeLists.txt
trunk/reactos/ntoskrnl/cache/cachesub.c
trunk/reactos/ntoskrnl/cache/copysup.c
trunk/reactos/ntoskrnl/cache/fssup.c
trunk/reactos/ntoskrnl/cache/lazyrite.c
trunk/reactos/ntoskrnl/cache/logsup.c
trunk/reactos/ntoskrnl/cache/mdlsup.c
trunk/reactos/ntoskrnl/cache/newcc.h
trunk/reactos/ntoskrnl/cache/pinsup.c
trunk/reactos/ntoskrnl/cache/section/data.c
trunk/reactos/ntoskrnl/cache/section/fault.c
trunk/reactos/ntoskrnl/cache/section/io.c
trunk/reactos/ntoskrnl/cache/section/newmm.h
trunk/reactos/ntoskrnl/cache/section/reqtools.c
trunk/reactos/ntoskrnl/cache/section/sptab.c
trunk/reactos/ntoskrnl/cache/section/swapout.c
trunk/reactos/ntoskrnl/cc/fs.c
trunk/reactos/ntoskrnl/include/internal/mm.h
trunk/reactos/ntoskrnl/mm/amd64/page.c
trunk/reactos/ntoskrnl/mm/i386/page.c
trunk/reactos/ntoskrnl/mm/marea.c
trunk/reactos/ntoskrnl/mm/mmfault.c
trunk/reactos/ntoskrnl/mm/mminit.c
trunk/reactos/ntoskrnl/mm/rmap.c
trunk/reactos/ntoskrnl/mm/section.c
trunk/reactos/ntoskrnl/ntoskrnl-generic.rbuild
trunk/reactos/ntoskrnl/po/poshtdwn.c
[This mail would be too long, it was shortened to contain the URLs only.]
Modified: trunk/reactos/ntoskrnl/CMakeLists.txt
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/CMakeLists.txt?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/cache/cachesub.c
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/cache/cachesub.c?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/cache/copysup.c
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/cache/copysup.c?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/cache/fssup.c
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/cache/fssup.c?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/cache/lazyrite.c
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/cache/lazyrite.c?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/cache/logsup.c
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/cache/logsup.c?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/cache/mdlsup.c
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/cache/mdlsup.c?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/cache/newcc.h
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/cache/newcc.h?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/cache/pinsup.c
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/cache/pinsup.c?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/cache/section/data.c
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/cache/section/data.c?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/cache/section/fault.c
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/cache/section/fault.c?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/cache/section/io.c
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/cache/section/io.c?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/cache/section/newmm.h
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/cache/section/newmm.h?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/cache/section/reqtools.c
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/cache/section/reqtools.c?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/cache/section/sptab.c
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/cache/section/sptab.c?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/cache/section/swapout.c
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/cache/section/swapout.c?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/cc/fs.c
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/cc/fs.c?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/include/internal/mm.h
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/include/internal/mm.h?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/mm/amd64/page.c
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/mm/amd64/page.c?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/mm/i386/page.c
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/mm/i386/page.c?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/mm/marea.c
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/mm/marea.c?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/mm/mmfault.c
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/mm/mmfault.c?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/mm/mminit.c
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/mm/mminit.c?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/mm/rmap.c
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/mm/rmap.c?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/mm/section.c
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/mm/section.c?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/ntoskrnl-generic.rbuild
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/ntoskrnl-generic.rbuild?rev=55833&r1=55832&r2=55833&view=diff
Modified: trunk/reactos/ntoskrnl/po/poshtdwn.c
URL:
http://svn.reactos.org/svn/reactos/trunk/reactos/ntoskrnl/po/poshtdwn.c?rev=55833&r1=55832&r2=55833&view=diff
