Hi - Despite the radio silence for the last few months, I've been busy on my ramfs branch, and it's done well enough to push it out.
Very short version: - The VFS is gone, 9ns now has mmap and everything else I needed, both for a ramfs and even over a 9p mount. We also have RCU. Relatively short version: - The old VFS is completely gone - All filesystem calls operate on the Plan 9 namespaces (9ns). You can even cd now! - There is a helper infrastructure called "tree files" that any device can use to define its in-memory file system. A tree_file is an fs_file plus the tree hierarchy. An fs_file is the elements of a file: e.g. the page cache, permission bits, etc. For those familiar with Plan 9, it contains the struct Dir. - The tree file code has all of the helpers for interfacing with Plan 9 devices, e.g. struct dev's functions: walk, open, create, read, write, etc. - The tree file code uses RCU whenever possible. For instance, if all of the tree_files are in memory, we can walk from / through several directories without locking intermediate files. - The 9ns code can handle symlinks, mmap (if the device supports fs_files) and atomic rename across directories (if the device uses tree_files). - The old KFS (which was a VFS plugin) is now #kfs - it is a tree_file system that populates from the initramfs, similar to the old KFS. - There is a #tmpfs that is another in-memory filesystem, like KFS, but it starts empty and deletes itself when you unmount and close all of its FDs. I mostly used this for debugging, but it's cool regardless. - The mount device can use a tree file system frontend as a cache. If you mount with -C, sys_mount will layer #gtfs (generic tree file system) in front of your specific #mnt service. #gtfs knows to talk to some underlying device, speaking 9p. #mnt itself has basically no changes. This means we can mmap and run binaries across 9p without reading the entire file into memory. - Here's how to think of #gtfs: #mnt takes a chan that speaks the 9p RPC interface and converts that to the in-kernel 9p function pointer interface, binding it to the namespace. #gtfs takes a chan that speaks the 9p *functional* interface (i.e. struct dev function pointers) and converts it to the tree_file interface, which is then bound into the namespace. - #gtfs is a write-back cache. By comparison, #kfs and #tmpfs hold the real data of a file. The tree_file code can handle both of these situations, and it can handle having threads remove elements of the cache. The ram FSs simply have a refcount to keep their files alive, and they don't worry about writeback. - I have a bunch of helpers to remove data from the page cache, LRU files from the tree file tree, and other things. For now, they aren't hooked into the memory subsystem, so the way to free up RAM is to unmount. There is a 'sync' program that will ensure data gets written back - e.g. you're about to crash, and want to make sure your data is sent across the 9p chan. - I mentioned RCU: we have it now. It's based off of Linux's tree RCU. Our variant is similar to rcu-sched, and quiescent states consist of idle, userspace, and in between routine kernel messages. The callbacks from call_rcu are queued on each core, but they are executed on management cores (core 0) to avoid interfering with the CG cores. We do not have softirq processing or any sort of IPI that might interfere with a core. Similar to Linux, we can detect remotely whether or not a core is in userspace or idle (similar to their dyntick idle checks). - There are a lot of bug fixes and whatnot. See the long version. Here's the long version. Pull to try it out and check the commit log. Barret The following changes since commit 4e8396e2d74370428d03f2428589171d6e33105b: vmm: Clean up the VM cmdline files (2017-12-22 14:15:04 -0500) are available in the Git repository at: g...@github.com:brho/akaros.git ramfs^ for you to fetch changes up to 54a1d3c18f8a556a2d4caf7f4c74b4682efd15a7: rcu: Implement RCU (2018-04-30 14:41:01 -0400) ---------------------------------------------------------------- View this online at: https://github.com/brho/akaros/compare/4e8396e2d743...54a1d3c18f8a ---------------------------------------------------------------- Barret Rhoden (113): Fix 'include' issue slab: Catch attempted NULL frees Add a couple #defines to stdint Remove the #defines for struct stat and dirent (XCC) x86: Add a kfunc for testing wrmsr() x86: Fix PTE_ bit definitions (XCC) proc: Move decref out of __proc_disown_child() proc: Fix buggy disowning of children mlx4: Disable the user-level bypass busybox: Clear errstr after setting errno Add an unmount helper vfs: Use a fake device number for KFS 9ns: Add helpers for permissions checks Move 'KFOP' helpers to umem.c devalarm: Use the dummy dev functions 9ns: Set eve's name to "eve" 9ns: Add 'extensions' to dev.create 9ns: Add 'extensions' to namec() 9ns: Extend struct dir and the stat M bufs 9ns: Add high-level support for symlinks 9ns: Use a helper struct for walk/namec options 9ns: Make walk() follow symlinks 9ns: Fix a couple namec() error cases 9ns: Sort out when to not follow symlinks 9ns: Allow binds on symlinks mm: Fix type bug mm: Stop leaking refs on mmap() errors mm: Fix permission checks mm: Ignore the FD on MAP_ANON mmaps mm: Use struct file_or_chan for VMR files mm: Call dev.mmap before checking the file mm: Clean up the functional interface mm: Allow dev.mmap() to block 9ns: Fix devtab function pointer signatures Add a garbage RCU implementation pm: Add a stub for pm_destroy() 9ns: Add fs_files and tree_files 9ns: Add error_jmp(): jump with an existing error Set O_EXEC when the kernel opens a file for exec mm: 9ns: Add support for mmap 9ns: Support SYS_access (XCC) 9ns: Support SYS_rmdir 9ns: Remove support for SYS_rename Add atomic_swap_ptr() 9ns: Implement chdir/fchdir 9ns: allow 'current-less' operations in 9ns Extract parse_cpio_entries() from KFS 9ns: Add #kfs 9ns: kfs: Add root entries 9ns: Remove #root 9ns: Add get_cwd() vfs: Remove syscall access to the VFS vfs: Remove ext2 support vfs: Remove fs_env from struct proc vfs: Remove KFS, blockdev and devfs vfs: Remove monitor functions vfs: Remove misc tests vfs: Remove the guts of the VFS vfs: Remove the last references to struct file * vfs: Move FD management code to 9ns vfs: Remove vfs.h kfs: Remove a few old debugging files pm: Track the fs_file instead of the inode/blockdev 9ns: Add #tmpfs 9ns: Support rename strace: Report errno and errstr on failure 9ns: Clean up the dir->mode bits 9ns: Remove the old mnt cache 9ns: Add the cache option (-C) to mount exec: Clear owning proc earlier 9ns: Make QTFILE == 0. Goddamn 9p 9ns: Add LRU support to tree filesystems mm: Mark dirty pages when we munmap 9ns: Allow racy truncations radix: Add for_each iterators radix: Implement radix_tree_destroy() pm: Implement pm_destroy() pm: Implement pm_remove_or_zero_pages() pm: Allow RCU lookups and qlocked updaters mm: Catch allocation failure Add pte_clear_dirty() mm: Don't decref files while holding a lock mm: Hold the pte_lock when doing a memwalk 9ns: Add MEM_WAIT in a few places Fix up bind and mount pm: Add pm_writeback_pages() pm: Add pm_free_unused_pages() pm: Remove the venerable pm_remove_contig() 9ns: Clarify a couple struct dev fields Use READ_ONCE() for seq counters (XCC) 9ns: Add #gtfs 9ns: kfs: Give the CPIO blob to the base arena 9ns: Add a helper for getting chan's device's name 9ns: Make chan_ctl() mandatory 9ns: Implement fsync with chan_ctl 9ns: Support setting FD flags (XCC) 9ns: mnt: Don't use a 'bogus' struct 9ns: Remove CCACHE Add pcpui helpers Move PASTE() to common.h Update Linux's list.h Copy Linux's compiler.h Use Linux's compiler headers (XCC) Fix "error macro requires 3 arguments" problem Remove ONE_xBYTE #defines sched: Catch the lack of a scheduler type Add a bulk interface to sem_down() Add arch/membar.h Allow PERCPU array types Fix some minor Linux spatch / conversions Add a few smp_ helpers from Linux. rcu: Add Linux headers and helpers rcu: Implement RCU Kconfig | 17 - Makefile | 15 +- config-default | 1 - kern/arch/riscv/arch.h | 2 +- kern/arch/riscv/atomic.h | 5 + kern/arch/riscv/membar.h | 10 + kern/arch/riscv/pmap_ops.h | 5 + kern/arch/riscv/types.h | 5 - kern/arch/x86/apic9.c | 2 - kern/arch/x86/atomic.h | 7 +- kern/arch/x86/devarch.c | 9 +- kern/arch/x86/ioapic.c | 2 - kern/arch/x86/kpt.h | 5 + kern/arch/x86/membar.h | 37 + kern/arch/x86/mp.c | 2 - kern/arch/x86/mpacpi.c | 2 - kern/arch/x86/msi.c | 2 - kern/arch/x86/perfmon.c | 1 - kern/arch/x86/pmap_ops.h | 6 + kern/arch/x86/rdtsc_test.c | 21 + kern/arch/x86/ros/mmu64.h | 25 +- kern/arch/x86/trap.c | 2 +- kern/arch/x86/types.h | 5 - kern/arch/x86/vmm/ept.h | 5 + kern/drivers/dev/Kbuild | 4 +- kern/drivers/dev/acpi.c | 10 +- kern/drivers/dev/alarm.c | 30 +- kern/drivers/dev/capability.c | 10 +- kern/drivers/dev/cons.c | 10 +- kern/drivers/dev/coreboot.c | 2 - kern/drivers/dev/ether.c | 18 +- kern/drivers/dev/eventfd.c | 8 +- kern/drivers/dev/gtfs.c | 851 ++++++ kern/drivers/dev/kfs.c | 339 +++ kern/drivers/dev/kprof.c | 9 +- kern/drivers/dev/mem.c | 8 +- kern/drivers/dev/mnt.c | 75 +- kern/drivers/dev/pci.c | 11 +- kern/drivers/dev/pipe.c | 29 +- kern/drivers/dev/proc.c | 36 +- kern/drivers/dev/random.c | 18 +- kern/drivers/dev/regress.c | 18 +- kern/drivers/dev/root.c | 470 --- kern/drivers/dev/sd.c | 30 +- kern/drivers/dev/sdiahci.c | 2 - kern/drivers/dev/sdscsi.c | 2 - kern/drivers/dev/srv.c | 24 +- kern/drivers/dev/tmpfs.c | 283 ++ kern/drivers/dev/vars.c | 17 +- kern/drivers/dev/version.c | 8 +- kern/drivers/net/Kbuild | 4 +- kern/drivers/net/bnx2x/bnx2x_dev.c | 2 - kern/drivers/net/ether8139.c | 2 - kern/drivers/net/ether82563.c | 2 - kern/drivers/net/etherigbe.c | 2 - kern/drivers/net/ethermii.c | 2 - kern/drivers/net/mlx4/main.c | 4 +- kern/drivers/net/udrvr/compat.c | 15 +- kern/drivers/net/udrvr/compat.h | 1 + kern/include/atomic.h | 1 + kern/include/blockdev.h | 98 - kern/include/common.h | 3 +- kern/include/compiler.h | 38 +- kern/include/corerequest.h | 2 + kern/include/cpio.h | 3 +- kern/include/crypto/2sysincludes.h | 1 - kern/include/devfs.h | 24 - kern/include/elf.h | 6 +- kern/include/env.h | 4 +- kern/include/err.h | 9 +- kern/include/ext2fs.h | 263 -- kern/include/fs_file.h | 134 + kern/include/kfs.h | 84 - kern/include/kthread.h | 6 + kern/include/linux/compat_todo.h | 63 +- kern/include/linux/compiler-clang.h | 17 + kern/include/linux/compiler-gcc.h | 339 +++ kern/include/linux/compiler-intel.h | 45 + kern/include/linux/compiler.h | 609 ++++ kern/include/linux_compat.h | 25 +- kern/include/list.h | 93 +- kern/include/mm.h | 54 +- kern/include/monitor.h | 2 - kern/include/ns.h | 252 +- kern/include/pagemap.h | 25 +- kern/include/percpu.h | 7 +- kern/include/process.h | 2 +- kern/include/profiler.h | 4 +- kern/include/radix.h | 37 +- kern/include/rbtree.h | 6 +- kern/include/rcu.h | 93 + kern/include/rcu_helper.h | 243 ++ kern/include/rcu_node_tree.h | 103 + kern/include/rculist.h | 673 +++++ kern/include/rcupdate.h | 518 ++++ kern/include/ros/atomic.h | 9 +- kern/include/ros/common.h | 16 +- kern/include/ros/fs.h | 11 +- kern/include/smp.h | 7 + kern/include/stdint.h | 12 + kern/include/syscall.h | 1 + kern/include/tree_file.h | 281 ++ kern/include/umem.h | 2 + kern/include/vfs.h | 526 ---- kern/kfs/dev/.empty | 0 kern/kfs/dir1/dir1-1/f1-1.txt | 1 - kern/kfs/dir1/f1.txt | 1 - kern/kfs/dir2/f2.txt | 1 - kern/kfs/kfs_test.txt | 2 - kern/kfs/mnt/.empty | 0 kern/kfs/mountroot | 2 +- kern/kfs/net/.empty | 0 kern/kfs/proc/.empty | 0 kern/kfs/prof/.empty | 0 kern/kfs/prog/.empty | 0 kern/kfs/srv/.empty | 0 kern/lib/Kbuild | 1 + kern/lib/cpio.c | 63 + kern/src/Kbuild | 6 +- kern/src/blockdev.c | 306 -- kern/src/devfs.c | 59 - kern/src/elf.c | 32 +- kern/src/err.c | 2 - kern/src/ext2fs.c | 1537 ---------- kern/src/fdtap.c | 1 - kern/src/init.c | 13 +- kern/src/kdebug.c | 4 +- kern/src/kfs.c | 971 ------- kern/src/ktest/pb_ktests.c | 270 +- kern/src/kthread.c | 50 +- kern/src/manager.c | 3 - kern/src/mm.c | 543 ++-- kern/src/monitor.c | 147 +- kern/src/net/arp.c | 2 - kern/src/net/devip.c | 33 +- kern/src/net/dial.c | 2 - kern/src/net/eipconv.c | 2 - kern/src/net/ethermedium.c | 2 - kern/src/net/icmp.c | 4 - kern/src/net/ip.c | 2 - kern/src/net/ipaux.c | 2 - kern/src/net/ipifc.c | 2 - kern/src/net/ipprotoinit.c | 2 - kern/src/net/iproute.c | 2 - kern/src/net/iprouter.c | 2 - kern/src/net/loopbackmedium.c | 2 - kern/src/net/netaux.c | 2 - kern/src/net/netif.c | 2 - kern/src/net/netlog.c | 2 - kern/src/net/nullmedium.c | 2 - kern/src/net/pktmedium.c | 2 - kern/src/net/plan9.c | 2 - kern/src/net/ptclbsum.c | 2 - kern/src/net/tcp.c | 2 - kern/src/net/udp.c | 4 - kern/src/ns/Kbuild | 3 +- kern/src/ns/allocb.c | 2 - kern/src/ns/cache.c | 66 - kern/src/ns/chan.c | 315 +- kern/src/ns/cleanname.c | 2 - kern/src/ns/convD2M.c | 70 +- kern/src/ns/convM2D.c | 141 +- kern/src/ns/convM2S.c | 20 +- kern/src/ns/convM2kdirent.c | 158 +- kern/src/ns/convS2M.c | 2 - kern/src/ns/dev.c | 65 +- kern/src/ns/fs_file.c | 497 ++++ kern/src/ns/getfields.c | 2 - kern/src/ns/parse.c | 2 - kern/src/ns/pgrp.c | 6 - kern/src/ns/qio.c | 2 - kern/src/ns/sysfile.c | 588 +++- kern/src/ns/tokenize.c | 2 - kern/src/ns/tree_file.c | 1349 +++++++++ kern/src/ns/util.c | 54 + kern/src/pagemap.c | 575 ++-- kern/src/process.c | 68 +- kern/src/profiler.c | 9 +- kern/src/radix.c | 212 +- kern/src/rcu.c | 602 ++++ kern/src/rcu_tree_helper.c | 263 ++ kern/src/schedule.c | 1 + kern/src/slab.c | 1 + kern/src/smp.c | 2 + kern/src/syscall.c | 772 ++--- kern/src/trap.c | 5 + kern/src/umem.c | 21 + kern/src/vfs.c | 3014 -------------------- scripts/etherboot/fixincludes | 1 - scripts/fixincludes | 1 - scripts/spatch/linux/funcs.cocci | 22 - scripts/spatch/linux/sync.cocci | 7 + tests/bind.c | 28 +- tests/mount.c | 38 +- tests/sync.c | 30 + tests/unmount.c | 45 + .../akaros-patches/1.17.3/0009-unset_errno.patch | 10 + 197 files changed, 10511 insertions(+), 9956 deletions(-) create mode 100644 kern/arch/riscv/membar.h create mode 100644 kern/arch/x86/membar.h create mode 100644 kern/drivers/dev/gtfs.c create mode 100644 kern/drivers/dev/kfs.c delete mode 100644 kern/drivers/dev/root.c create mode 100644 kern/drivers/dev/tmpfs.c delete mode 100644 kern/include/blockdev.h delete mode 100644 kern/include/devfs.h delete mode 100644 kern/include/ext2fs.h create mode 100644 kern/include/fs_file.h delete mode 100644 kern/include/kfs.h create mode 100644 kern/include/linux/compiler-clang.h create mode 100644 kern/include/linux/compiler-gcc.h create mode 100644 kern/include/linux/compiler-intel.h create mode 100644 kern/include/linux/compiler.h create mode 100644 kern/include/rcu.h create mode 100644 kern/include/rcu_helper.h create mode 100644 kern/include/rcu_node_tree.h create mode 100644 kern/include/rculist.h create mode 100644 kern/include/rcupdate.h create mode 100644 kern/include/tree_file.h delete mode 100644 kern/include/vfs.h create mode 100644 kern/kfs/dev/.empty delete mode 100644 kern/kfs/dir1/dir1-1/f1-1.txt delete mode 100644 kern/kfs/dir1/f1.txt delete mode 100644 kern/kfs/dir2/f2.txt delete mode 100644 kern/kfs/kfs_test.txt create mode 100644 kern/kfs/mnt/.empty create mode 100644 kern/kfs/net/.empty create mode 100644 kern/kfs/proc/.empty create mode 100644 kern/kfs/prof/.empty create mode 100644 kern/kfs/prog/.empty create mode 100644 kern/kfs/srv/.empty create mode 100644 kern/lib/cpio.c delete mode 100644 kern/src/blockdev.c delete mode 100644 kern/src/devfs.c delete mode 100644 kern/src/ext2fs.c delete mode 100644 kern/src/kfs.c delete mode 100644 kern/src/ns/cache.c create mode 100644 kern/src/ns/fs_file.c create mode 100644 kern/src/ns/tree_file.c create mode 100644 kern/src/rcu.c create mode 100644 kern/src/rcu_tree_helper.c create mode 100644 tests/sync.c create mode 100644 tests/unmount.c create mode 100644 tools/apps/busybox/akaros-patches/1.17.3/0009-unset_errno.patch -- You received this message because you are subscribed to the Google Groups "Akaros" group. To unsubscribe from this group and stop receiving emails from it, send an email to akaros+unsubscr...@googlegroups.com. To post to this group, send email to akaros@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
