Nadav, Thanks for your feedback! Very much appreciated as always! On Wednesday, November 14, 2018 at 2:53:06 AM UTC-5, Nadav Har'El wrote: > > Thanks! Nice plan. I have few small comments below. > > On Tue, Nov 13, 2018 at 8:54 PM Waldek Kozaczuk <[email protected] > <javascript:>> wrote: > >> This is somewhat revised version of the roadmap I sent couple of months >> ago. I think that the main premise of it which I wrote in the original >> version still stands: >> >> "In my opinion, if OSv is to become more relevant than it is now, its >> primary target should be a platform for running stateless and serverless >> apps. Therefore it needs to become leaner (memory usage and kernel size), a >> little more modular and boot faster while preserving its functionality in >> terms of Linux/POSIX ABI (and possibly expanding where it makes sense). >> That is not say that we should not improve support for stateful apps like >> mysql or elasticsearch which would require optimizations to VFS and >> possibly ZFS which I list below." >> >> *OBJECTIVES* >> >> - Make OSv more modular >> - >> - ZFS and commands/runscript functionality should be moved as >> optional modules in form of shared libraries >> >> Because this adds complexity (to the build system, to the images, etc.) > we should probably investigate the benefits (in image size, memory size, > etc.) before actually doing this. >
Moving ZFS code outside by itself reduces kernel size by 800-900K. Program options is another 500-600K. This means faster boot and less memory used and less code if not needed. > > Also, in the case of ZFS, I wonder if there's a benefit of doing ZFS as a > shared object or just have a build option for enabling and disabling it in > the main kernel. I.e., if someone uses scripts/build to build an image with > ZFS, or capstan which always assumes ZFS, we can compile the kernel with > ZFS. > The build option is the easiest way but then we would end up with multiple version of kernel. But it could be the good first step. The ideal solution would be to load ZFS shared library from ROFS. I recently upstreamed changes to capstan to allow building ROFS images. > > About the commands thing: Personally I think command-line parsing is so > trivial that it would be sad to take it out just because we used bloated > Boost features (Qi and Options). More on this below. > I do not want to remove commands logic completely. Instead I want to make it optional (no needed, I think, to run single app, right?). I think most complexity goes into supporting runscript. In theory we could create our simple small version of commands.cc:parse_command_line(const std::string line, bool& ok) using getopt_long(). > > >> >> - >> - The key building block to achieve it should be an improved ROFS >> intended as a default filesystem to load code from >> - Optimize kernel size >> - >> - My experiments show that kernel (loader-stripped.elf) can be as >> small as 5.1MB after ZFS and program options are compiled out (and >> there is >> still room for improvements) >> - Optimize boot time >> - >> - OSv can execute under 50ms on hyperkit and under 100ms on QEMU >> with qboot >> - Smaller kernel should reduce boot time >> - Optimize memory usage >> - >> - My recent investigation about memory allocation is OSv shows >> that it should be possible and quite easy to reduce memory utilization >> in >> many areas >> - The aggressive but quite realistic goal should be an ability to >> run OSv with simple C app using *no more than 10MB of RAM* >> - Smaller kernel should also reduce memory usage >> >> >> MORE IMPORTANT >> File systems >> >> - ROFS >> - >> - [FASTER BOOT] Add compression - >> https://github.com/cloudius-systems/osv/issues/978 >> - [LESS MEMORY] Avoid creating another copy when mmap-ing ROFS >> file - https://github.com/cloudius-systems/osv/issues/979 >> - ZFS >> - >> - [MODULARITY, SMALLER KERNEL, FASTER BOOT] move all ZFS code in >> kernel into libzfs.so >> - >> - Make ZFS an optional library - >> https://github.com/cloudius-systems/osv/issues/1009 >> - RAMFS >> - >> - [SPEED, LESS MEMORY] Fix slow write/append of files on RAMFS - >> https://github.com/cloudius-systems/osv/issues/884 >> - Eliminate BootFS >> >> You want to eliminate this filesystem (why?) or some of the files we now > put in it by default? > I might not be clear here. I do NOT want to eliminate ramfs or bootfs (ramfs really) when used to create ramfs-only image. There are use cases where it make sense. However I do want to replace bootfs with rofs (see https://github.com/cloudius-systems/osv/issues/1009 for more details). As it is now ZFS images require some code in bootfs (mkfs.so, etc) which then gets duplicated. Bootfs in in this case makes kernel bigger, takes more memory, etc. Loading code from ROFS almost always makes more sense especially once we make it compress-able. > >> - 9pFS >> - >> - [ENHANCEMENTS] >> https://github.com/cloudius-systems/osv/issues/1008 >> >> >> Optimize memory usage >> >> - Wasted memory in early (pre-SMP-enabled) malloc (*possibly save 6MB >> of memory!!!*) - https://github.com/cloudius-systems/osv/issues/270 >> - Make allocations < 16 bytes more space efficient - >> https://github.com/cloudius-systems/osv/issues/1011 >> >> This is a really low hanging fruit that should definitely be fixed :-) > > >> >> - >> - Improve physical memory utilization by using memory below 2MB - >> https://github.com/cloudius-systems/osv/issues/1012 >> - Make L1/L2 memory pool sizes self-configurable depending on >> physical memory available - >> https://github.com/cloudius-systems/osv/issues/1013 >> - Consider more space-efficient allocation between 1024 and 4096 >> bytes - https://github.com/cloudius-systems/osv/issues/1000 >> >> >> Improve modularity >> >> - Remove boost programs_options from kernel by rewriting >> loader.cc::parse_options() to use getopt_long() and extract >> commands.cc:parse_command_line(const std::string line, bool& ok) as a >> optional library commands.so; see >> https://github.com/cloudius-systems/osv/issues/1014 for details >> >> I think moving the parsing - both the boost options and the boost Qi > (used for parsing ";", quotes, etc. in the command line) to a separate > shared object, indeed makes sense as an easy first step. It means OSv can > load this library, parse the options, and then unload it. > However, I'm worried we have a chicken-and-egg problem in the sense that > we want to store this library in ZFS not in bootfs (otherwise we don't save > any memory), and we only mount ZFS after we parse the kernel's options? > We will not have this problem if we load commands.so from ROFS. > > I wonder if it wouldn't make sense to just rewrite the command line > parsing using getopt_long() plus perhaps (if it saves code size...) a > simple state machine for the Qi stuff, and keep it in the kernel after it > becomes a tiny piece of code. But maybe it's not worth the effort... > > >> >> - >> >> >> Optimize kernel size >> >> - Be more selective on symbols exported from the kernel - >> https://github.com/cloudius-systems/osv/issues/97 >> - >> - Remove/hide symbols >> - Investigate what else can be discarded using bloaty >> - >> - eliminate some debug strings in BSD code tree that should >> further reduce .rodata section >> >> > What "debug strings"? > I am referring to the strings used by KASSERT macro (bsd code) and similar macros that contribute to rodata size. We could make those strings disappear in the release version of kernel (line number should be enough right?), the debug one could still have it. I have not measured how much it is. So it may not be that important. > >> - >> - remove/hide symbols (reduces ELF .dynsym (others?) section) - >> https://github.com/cloudius-systems/osv/issues/9 >> <https://github.com/cloudius-systems/osv/issues/97>7 >> - reduce size of other sections in loader.elf >> >> >> Other >> >> - Make more programs run without having to re-compile (on ubuntu >> already pies) >> - >> - look at adding missing symbols to make coreutils (ls, find) >> programs run (sent email about it) >> - Stop using libraries and headers from external - >> https://github.com/cloudius-systems/osv/issues/743 and related … >> https://github.com/cloudius-systems/osv/issues/687 >> - Assertion failed: timestamp >= _last (timer-set.hh: expire) - >> https://github.com/cloudius-systems/osv/issues/382 >> >> I would really love for this 4 year old bug to go away, but unfortunately > could never figure it out. > I came across an interesting observation - that the jump is always exactly > 9 * 2^32 ns, and only specific to HPET - but was never able to explain why > it happens. > > >> >> - >> >> >> COMPLETE BUT NOT COMMITTED PATCHES >> >> - [SECURITY] Read-only REST API >> - >> - https://github.com/cloudius-systems/osv/issues/820 >> - mprotect() syscall stack >> - >> - we could commit it as is but then it would syscall stack use >> more memory (1 page per thread) until we fix this issue - >> https://github.com/cloudius-systems/osv/issues/1000 >> - full revival of multiboot >> - >> - there may be still controversial issues resolved before this >> patch is ready to be merged >> >> >> LESS IMPORTANT >> >> - [ENHANCEMENTS] Support new hypervisors >> - >> - qboot >> - NEMU >> - Hyperkit - https://github.com/cloudius-systems/osv/issues/948 >> - >> - multiboot - https://github.com/cloudius-systems/osv/issues/981 >> - File systems >> - >> - RAMFS >> - >> - [LESS MEMORY] Avoid creating another copy when mmap-ing RAMF >> - https://github.com/cloudius-systems/osv/issues/979 >> - [BUG] readdir does not handle asynchronous entry removal >> properly - https://github.com/cloudius-systems/osv/issues/68 >> - ZFS >> - >> - [ENHANCEMENTS] create ZFS image on host instead of booting >> OSv - see https://github.com/cloudius-systems/osv/issues/918 >> - Make VFS locking more granular >> - >> - VFS: read()/write() lock the vnode for too long - >> https://github.com/cloudius-systems/osv/issues/450 >> - >> - this commit addresses ZFS - >> >> https://github.com/cloudius-systems/osv/commit/b5eadc37f12c4b97a52705830d2d9097498049c2 >> >> - >> - people reported that some FS apps did not scale even >> though commit above addresses >> - this commit >> >> https://github.com/cloudius-systems/osv/commit/697943f631960b3d55f66381581e1435b726a1a8 >> reverted >> previous one because of >> https://github.com/cloudius-systems/osv/issues/504 >> - Async IO - >> https://github.com/cloudius-systems/osv/issues/656 >> - [MONITORING] REST API enhancements >> - >> - Remote test execution - >> https://github.com/cloudius-systems/osv/issues/445 >> - Page in/page out - >> https://github.com/cloudius-systems/osv/issues/465 >> - Block IO - https://github.com/cloudius-systems/osv/issues/466 >> - Network stat - https://github.com/cloudius-systems/osv/issues/467 >> - Load average - https://github.com/cloudius-systems/osv/issues/469 >> - [SECURITY] OpenSSL upgrade >> - Do not use setimg.py in Makefile - >> https://github.com/cloudius-systems/osv/issues/733 >> >> >> IN PROGRESS >> >> - Add support for Mono, C# - >> https://github.com/cloudius-systems/osv/issues/34 >> - >> - added sample app >> >> -- >> You received this message because you are subscribed to the Google Groups >> "OSv Development" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
