Hi -
I finished the overall of the memory allocator and related parts.
Major highlights:
- Uses an arena allocator, based on Bonwick et al's vmem allocator, for
all page allocations. This greatly speeds up our contiguous memory
allocations. If you ever had 'ps' take a long time after you
allocated a GB of RAM or so, this was the culprit.
- The slab allocator (kmem caches) has a per-core magazine layer,
again based off Bonwick's magazine/depot layer. This reduces global
lock contention for all slab allocators, including kernel messages
and kernel memory allocations of common sizes.
- The arena allocators, notable kpages_arena, can have slab allocators
inside of them. Grep qcache in arena.c for details.
- The kernel virtual address mappings, used for things like MMIO, but
not for the KERNBASE mapping (0xffff8000.00000000 -> 0) is managed by
an arena allocator. Previously it was a home-brewed incremental
allocator. It actually nests two arenas, using the interposition of
the 'free func' to batch the actual freeing of old mappings. This
allows me to amortize the cost of the global TLB shootdown.
- Kernel stacks are now mapped with a virtual address mapping and have
guard pages at the end. This removes a source of bugs and paranoia:
running off the end of your stack. Stack pointers of this variety now
look like e.g. 0xfffffff000242fd8. If you run off the end of the
stack, you'll get a double fault and panic.
- Reclaim isn't set up yet, though there's some infrastructure in place
for it.
- Memory diagnostics are available at #mem. Common things to cat are
'free' and 'kmemstat':
$ cat \#mem/free
Total Memory : 33026473984
Used Memory : 1179164672
Free Memory : 31847309312
$ cat \#mem/kmemstat
Arena/Slab Name :Typ: Objsize: Total Amt: Alloc Amt:
Allocs Ever
----------------------------------------------------------------------------------------
base : A : 4096: 33026473984: 1179164672:
12270
kpages : A : 4096: 1161715712: 1161715712:
9096
kpages_4096 : S : 4096: 1144258560: 1144258560:
499433
kpages_8192 : S : 8192: 7864320: 7782400:
78638
kpages_12288 : S : 12288: 122880: 36864:
2736
kpages_16384 : S : 16384: 393216: 376832:
0
kpages_20480 : S : 20480: 245760: 204800:
270
kpages_24576 : S : 24576: 122880: 49152:
0
kpages_28672 : S : 28672: 0: 0:
0
kpages_32768 : S : 32768: 131072: 32768:
0
kmalloc_64 : S : 64: 40320: 39552:
79601
kmalloc_128 : S : 128: 190464: 187008:
7508
kmalloc_256 : S : 256: 322560: 321792:
573833
kmalloc_512 : S : 512: 379904: 377856:
58529
kmalloc_1024 : S : 1024: 98304: 92160:
414
kmalloc_2048 : S : 2048: 360448: 346112:
18203
hash_entry : S : 32: 16128: 14624:
274
radix_nodes : S : 536: 88440: 85760:
0
kthread : S : 224: 32256: 31808:
4558
vm_regions : S : 80: 24000: 22240:
16483
struct_file : S : 144: 0: 0:
0
kernel_msgs : S : 64: 12096: 10816:
224563
dentry : S : 208: 173888: 172640:
5215
inode : S : 344: 124872: 123840:
0
file : S : 144: 8064: 5040:
1208
kfs_ino_info : S : 32: 12096: 11520:
0
proc : S : 2816: 45056: 25344:
269
block_reqs : S : 160: 0: 0:
0
buffer_heads : S : 56: 423360: 421848:
0
kmem_magazine : S : 512: 1107456: 1106944:
0
kmem_cache : S : 784: 15680: 15680:
0
kmem_slab : S : 40: 355520: 354840:
0
kmem_bufctl : S : 32: 8995392: 8993280:
0
vmap_addr : A : 4096: 66571993088: 2543616:
31
vmap : A : 4096: 2539520: 2539520:
31
kstack : S : 12288: 2359296: 2334720:
4558
I've run this on a few different machines, including our usual tests of
running a VM and ssh. My vmm branch is also sitting on top of mm, so
if you test that (which has some nasty, not-for-mainline hacks for
virtualization), then you'll also test the memory managers.
Let me know if you find any problems.
Barret
The following changes since commit 19fd49b2e99e0cc522390398c46557d149483230:
dune: clean up and remove lots of cruft (2016-11-21 19:23:55 -0500)
are available in the git repository at:
[email protected]:brho/akaros.git mm
for you to fetch changes up to 637cb6d93311c503bcb792ab6e30775070143234:
Remove CONFIG_KTHREAD_POISON (2016-11-28 14:47:26 -0500)
----------------------------------------------------------------
View this online at:
https://github.com/brho/akaros/compare/19fd49b2e99e...637cb6d93311
----------------------------------------------------------------
Barret Rhoden (66):
Fix the remaining /dev/ -> /dev_vfs/
Make page_insert() consume the caller's refcnt
Fix extra decref of shared_page
Remove SYS_cache_buster (XCC)
Fix bounds checks and misc errors in mm.c
Refactor map_page_at_addr
Provide a shim layer for reference counted pages
Remove page refcnts
Remove page coloring
Remove get_cont_phys_pages_at()
Move __always_inline to compiler.h
Import rbtrees from Linux
Integrate rbtrees into Akaros
Use a helper for resetting kernel stacks
x86: set pcpui->{ts,gdt} early
Jump to a real kstack ASAP during boot
x86: Stop freeing the trampoline page
mlx4: Remove page_is_free() safety check
Remove mon_gfp()
Add a #define for all MEM_FLAGS
Import hash.h from Linux
Port hash.h
Add hash_helper.h for custom dynamic hash tables
Add the arena allocator
Replace the old page allocator with the base arena
Moving 'booting' to a header
Check booting during trace_printk()
Set num_cores early in boot
slab: Use BSD_LISTs for the bufctls
slab: Bootstrap more kmem_caches
slab: Use a hashtable when looking up bufctls
slab: Add an arena pointer to the interface
slab: Support 'no-touch' caches
slab: Import resources from a source arena
slab: Move the name into the kmem_cache
slab: Bootstrap before setting up kpages_arena
arena: Use qcaches (slabs) in the arena
Remove kmalloc caches above PGSIZE
slab: Move ctors/dtors to the slab layer
slab: Stop appending a uintptr_t to small objects
slab: Remove obj_size from struct kmem_slab
x86: Pretend to be core 0 in smp_main()
Add spin_trylock_irqsave()
slab: Add the magazine and depot layers
Move assert in sem_down()
Put the size in the name of kmalloc caches
arena: Connecting importers with sources
Tracks arenas and slabs on tailqs
Remove old memory tests
Add #mem, for memory diagnostics
slab: Update the ctor/dtor interface
Convert calls of get_cont_pages() to kpages_alloc
Allocate natural alignment with get_cont_pages()
vmap: Handle unaligned vaddrs on vunmap_vmem()
vmap: Add a helper for global TLB shootdowns
vmap: Use an arena allocator for kernel vmaps
x86: Use global PTEs for kernel mappings
x86: Add EPTs to the boot_pmls
x86: Fix integer overflow in pml_for_each()
vmap: Make kernel intermediate mappings permanent
vmap: Use {map,unmap}_segment() helpers
x86: Put the boot PMLs and stacks in BSS
Fix backtrace_list()'s wild read
Use guard pages and KMC allocator for kstacks
x86: Handle double faults
Remove CONFIG_KTHREAD_POISON
Kconfig | 27 -
config-default | 7 -
kern/arch/riscv/Kbuild | 1 -
kern/arch/riscv/arch.h | 7 +
kern/arch/riscv/colored_caches.c | 38 --
kern/arch/riscv/colored_page_alloc.h | 48 --
kern/arch/riscv/page_alloc.c | 50 +-
kern/arch/riscv/pmap.c | 18 +
kern/arch/riscv/ros/mmu.h | 5 +-
kern/arch/x86/Kbuild | 1 -
kern/arch/x86/arch.h | 23 +-
kern/arch/x86/colored_caches.c | 51 --
kern/arch/x86/colored_page_alloc.h | 48 --
kern/arch/x86/cpuinfo.c | 5 +-
kern/arch/x86/entry64.S | 31 +-
kern/arch/x86/kdebug.c | 7 +-
kern/arch/x86/kernel64.ld | 2 +-
kern/arch/x86/page_alloc.c | 221 ++----
kern/arch/x86/pmap64.c | 97 ++-
kern/arch/x86/ros/mmu64.h | 22 +-
kern/arch/x86/ros/vmx.h | 1 -
kern/arch/x86/smp_boot.c | 32 +-
kern/arch/x86/topology.c | 8 +
kern/arch/x86/trap.c | 31 +-
kern/arch/x86/trap64.h | 3 +-
kern/arch/x86/trapentry64.S | 59 +-
kern/arch/x86/vmm/intel/vmx.c | 15 +-
kern/drivers/dev/Kbuild | 1 +
kern/drivers/dev/acpi.c | 80 ++-
kern/drivers/dev/kprof.c | 3 +-
kern/drivers/dev/mem.c | 461 +++++++++++++
kern/drivers/net/bnx2x/bnx2x_cmn.c | 2 +
kern/drivers/net/mlx4/en_rx.c | 30 +-
kern/drivers/net/mlx4/mlx4_en.h | 2 +-
kern/drivers/net/udrvr/compat.c | 12 +-
kern/drivers/net/udrvr/compat.h | 2 +-
kern/include/acpi.h | 1 +
kern/include/arena.h | 135 ++++
kern/include/atomic.h | 16 +
kern/include/colored_caches.h | 75 --
kern/include/colored_page_alloc.h | 68 --
kern/include/compiler.h | 2 +
kern/include/env.h | 5 -
kern/include/hash.h | 104 +++
kern/include/hash_helper.h | 57 ++
kern/include/init.h | 2 +
kern/include/kmalloc.h | 5 +-
kern/include/kthread.h | 4 +-
kern/include/linux/compat_todo.h | 26 +-
kern/include/linux_compat.h | 20 +-
kern/include/mm.h | 14 +-
kern/include/monitor.h | 1 -
kern/include/page_alloc.h | 27 +-
kern/include/pmap.h | 5 +
kern/include/rbtree.h | 130 ++++
kern/include/rbtree_augmented.h | 259 +++++++
kern/include/refd_pages.h | 57 ++
kern/include/ros/bits/syscall.h | 9 +-
kern/include/slab.h | 88 ++-
kern/lib/Kbuild | 1 +
kern/lib/rbtree.c | 601 ++++++++++++++++
kern/src/Kbuild | 2 +-
kern/src/arena.c | 1242 ++++++++++++++++++++++++++++++++++
kern/src/blockdev.c | 14 +-
kern/src/colored_caches.c | 283 --------
kern/src/env.c | 9 +-
kern/src/ext2fs.c | 6 +-
kern/src/frontend.c | 3 +-
kern/src/hashtable.c | 6 +-
kern/src/init.c | 20 +-
kern/src/kfs.c | 6 +-
kern/src/kmalloc.c | 22 +-
kern/src/ktest/pb_ktests.c | 202 ------
kern/src/kthread.c | 160 +++--
kern/src/manager.c | 1 -
kern/src/mm.c | 229 ++++---
kern/src/monitor.c | 21 -
kern/src/page_alloc.c | 402 ++---------
kern/src/pmap.c | 80 ++-
kern/src/printf.c | 2 +-
kern/src/printfmt.c | 3 -
kern/src/process.c | 28 +-
kern/src/radix.c | 6 +-
kern/src/slab.c | 733 +++++++++++++++-----
kern/src/smp.c | 14 +-
kern/src/syscall.c | 81 ---
kern/src/trap.c | 3 +-
kern/src/vfs.c | 9 +-
scripts/spatch/linux/memory.cocci | 2 +-
89 files changed, 4551 insertions(+), 2201 deletions(-)
delete mode 100644 kern/arch/riscv/colored_caches.c
delete mode 100644 kern/arch/riscv/colored_page_alloc.h
delete mode 100644 kern/arch/x86/colored_caches.c
delete mode 100644 kern/arch/x86/colored_page_alloc.h
create mode 100644 kern/drivers/dev/mem.c
create mode 100644 kern/include/arena.h
delete mode 100644 kern/include/colored_caches.h
delete mode 100644 kern/include/colored_page_alloc.h
create mode 100644 kern/include/hash.h
create mode 100644 kern/include/hash_helper.h
create mode 100644 kern/include/rbtree.h
create mode 100644 kern/include/rbtree_augmented.h
create mode 100644 kern/include/refd_pages.h
create mode 100644 kern/lib/rbtree.c
create mode 100644 kern/src/arena.c
delete mode 100644 kern/src/colored_caches.c
--
You received this message because you are subscribed to the Google Groups
"Akaros" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
For more options, visit https://groups.google.com/d/optout.
