On Wednesday, November 14, 2018 at 10:26:53 AM UTC-5, Waldek Kozaczuk wrote: > > Nadav, > Thanks for your feedback! Very much appreciated as always! > > On Wednesday, November 14, 2018 at 2:53:06 AM UTC-5, Nadav Har'El wrote: >> >> Thanks! Nice plan. I have few small comments below. >> >> On Tue, Nov 13, 2018 at 8:54 PM Waldek Kozaczuk <[email protected]> >> wrote: >> >>> This is somewhat revised version of the roadmap I sent couple of months >>> ago. I think that the main premise of it which I wrote in the original >>> version still stands: >>> >>> "In my opinion, if OSv is to become more relevant than it is now, its >>> primary target should be a platform for running stateless and serverless >>> apps. Therefore it needs to become leaner (memory usage and kernel size), a >>> little more modular and boot faster while preserving its functionality in >>> terms of Linux/POSIX ABI (and possibly expanding where it makes sense). >>> That is not say that we should not improve support for stateful apps like >>> mysql or elasticsearch which would require optimizations to VFS and >>> possibly ZFS which I list below." >>> >>> *OBJECTIVES* >>> >>> - Make OSv more modular >>> - >>> - ZFS and commands/runscript functionality should be moved as >>> optional modules in form of shared libraries >>> >>> Because this adds complexity (to the build system, to the images, etc.) >> we should probably investigate the benefits (in image size, memory size, >> etc.) before actually doing this. >> > > Moving ZFS code outside by itself reduces kernel size by 800-900K. Program > options is another 500-600K. This means faster boot and less memory used > and less code if not needed. > >> >> Also, in the case of ZFS, I wonder if there's a benefit of doing ZFS as a >> shared object or just have a build option for enabling and disabling it in >> the main kernel. I.e., if someone uses scripts/build to build an image with >> ZFS, or capstan which always assumes ZFS, we can compile the kernel with >> ZFS. >> > The build option is the easiest way but then we would end up with multiple > version of kernel. But it could be the good first step. The ideal solution > would be to load ZFS shared library from ROFS. I recently upstreamed > changes to capstan to allow building ROFS images. > >> >> About the commands thing: Personally I think command-line parsing is so >> trivial that it would be sad to take it out just because we used bloated >> Boost features (Qi and Options). More on this below. >> > I do not want to remove commands logic completely. Instead I want to make > it optional (no needed, I think, to run single app, right?). I think most > complexity goes into supporting runscript. In theory we could create our > simple small version of commands.cc:parse_command_line(const std::string > line, bool& ok) using getopt_long(). > >> >> >>> >>> - >>> - The key building block to achieve it should be an improved ROFS >>> intended as a default filesystem to load code from >>> - Optimize kernel size >>> - >>> - My experiments show that kernel (loader-stripped.elf) can be as >>> small as 5.1MB after ZFS and program options are compiled out (and >>> there is >>> still room for improvements) >>> - Optimize boot time >>> - >>> - OSv can execute under 50ms on hyperkit and under 100ms on QEMU >>> with qboot >>> - Smaller kernel should reduce boot time >>> - Optimize memory usage >>> - >>> - My recent investigation about memory allocation is OSv shows >>> that it should be possible and quite easy to reduce memory >>> utilization in >>> many areas >>> - The aggressive but quite realistic goal should be an ability to >>> run OSv with simple C app using *no more than 10MB of RAM* >>> - Smaller kernel should also reduce memory usage >>> >>> >>> MORE IMPORTANT >>> File systems >>> >>> - ROFS >>> - >>> - [FASTER BOOT] Add compression - >>> https://github.com/cloudius-systems/osv/issues/978 >>> - [LESS MEMORY] Avoid creating another copy when mmap-ing ROFS >>> file - https://github.com/cloudius-systems/osv/issues/979 >>> - ZFS >>> - >>> - [MODULARITY, SMALLER KERNEL, FASTER BOOT] move all ZFS code in >>> kernel into libzfs.so >>> - >>> - Make ZFS an optional library - >>> https://github.com/cloudius-systems/osv/issues/1009 >>> - RAMFS >>> - >>> - [SPEED, LESS MEMORY] Fix slow write/append of files on RAMFS - >>> https://github.com/cloudius-systems/osv/issues/884 >>> - Eliminate BootFS >>> >>> You want to eliminate this filesystem (why?) or some of the files we now >> put in it by default? >> > I might not be clear here. I do NOT want to eliminate ramfs or bootfs > (ramfs really) when used to create ramfs-only image. There are use cases > where it make sense. However I do want to replace bootfs with rofs (see > https://github.com/cloudius-systems/osv/issues/1009 for more details). As > it is now ZFS images require some code in bootfs (mkfs.so, etc) which then > gets duplicated. Bootfs in in this case makes kernel bigger, takes more > memory, etc. Loading code from ROFS almost always makes more sense > especially once we make it compress-able. > >> >>> - 9pFS >>> - >>> - [ENHANCEMENTS] >>> https://github.com/cloudius-systems/osv/issues/1008 >>> >>> >>> Optimize memory usage >>> >>> - Wasted memory in early (pre-SMP-enabled) malloc (*possibly save >>> 6MB of memory!!!*) - >>> https://github.com/cloudius-systems/osv/issues/270 >>> - Make allocations < 16 bytes more space efficient - >>> https://github.com/cloudius-systems/osv/issues/1011 >>> >>> This is a really low hanging fruit that should definitely be fixed :-) >> >> >>> >>> - >>> - Improve physical memory utilization by using memory below 2MB - >>> https://github.com/cloudius-systems/osv/issues/1012 >>> - Make L1/L2 memory pool sizes self-configurable depending on >>> physical memory available - >>> https://github.com/cloudius-systems/osv/issues/1013 >>> - Consider more space-efficient allocation between 1024 and 4096 >>> bytes - https://github.com/cloudius-systems/osv/issues/1000 >>> >>> >>> Improve modularity >>> >>> - Remove boost programs_options from kernel by rewriting >>> loader.cc::parse_options() to use getopt_long() and extract >>> commands.cc:parse_command_line(const std::string line, bool& ok) as >>> a optional library commands.so; see >>> https://github.com/cloudius-systems/osv/issues/1014 for details >>> >>> I think moving the parsing - both the boost options and the boost Qi >> (used for parsing ";", quotes, etc. in the command line) to a separate >> shared object, indeed makes sense as an easy first step. It means OSv can >> load this library, parse the options, and then unload it. >> However, I'm worried we have a chicken-and-egg problem in the sense that >> we want to store this library in ZFS not in bootfs (otherwise we don't save >> any memory), and we only mount ZFS after we parse the kernel's options? >> > We will not have this problem if we load commands.so from ROFS. > One of the objectives listed above: "The key building block to achieve it should be an improved ROFS intended as a default filesystem to load code from" is directly related to this. Is it a right move in right direction? I wonder what your thoughts are.
> >> I wonder if it wouldn't make sense to just rewrite the command line >> parsing using getopt_long() plus perhaps (if it saves code size...) a >> simple state machine for the Qi stuff, and keep it in the kernel after it >> becomes a tiny piece of code. But maybe it's not worth the effort... >> >> >>> >>> - >>> >>> >>> Optimize kernel size >>> >>> - Be more selective on symbols exported from the kernel - >>> https://github.com/cloudius-systems/osv/issues/97 >>> - >>> - Remove/hide symbols >>> - Investigate what else can be discarded using bloaty >>> - >>> - eliminate some debug strings in BSD code tree that should >>> further reduce .rodata section >>> >>> >> What "debug strings"? >> > I am referring to the strings used by KASSERT macro (bsd code) and similar > macros that contribute to rodata size. We could make those strings > disappear in the release version of kernel (line number should be enough > right?), the debug one could still have it. I have not measured how much it > is. So it may not be that important. > >> >>> - >>> - remove/hide symbols (reduces ELF .dynsym (others?) section) - >>> https://github.com/cloudius-systems/osv/issues/9 >>> <https://github.com/cloudius-systems/osv/issues/97>7 >>> - reduce size of other sections in loader.elf >>> >>> >>> Other >>> >>> - Make more programs run without having to re-compile (on ubuntu >>> already pies) >>> - >>> - look at adding missing symbols to make coreutils (ls, find) >>> programs run (sent email about it) >>> - Stop using libraries and headers from external - >>> https://github.com/cloudius-systems/osv/issues/743 and related … >>> https://github.com/cloudius-systems/osv/issues/687 >>> - Assertion failed: timestamp >= _last (timer-set.hh: expire) - >>> https://github.com/cloudius-systems/osv/issues/382 >>> >>> I would really love for this 4 year old bug to go away, but >> unfortunately could never figure it out. >> I came across an interesting observation - that the jump is always >> exactly 9 * 2^32 ns, and only specific to HPET - but was never able to >> explain why it happens. >> >> >>> >>> - >>> >>> >>> COMPLETE BUT NOT COMMITTED PATCHES >>> >>> - [SECURITY] Read-only REST API >>> - >>> - https://github.com/cloudius-systems/osv/issues/820 >>> - mprotect() syscall stack >>> - >>> - we could commit it as is but then it would syscall stack use >>> more memory (1 page per thread) until we fix this issue - >>> https://github.com/cloudius-systems/osv/issues/1000 >>> - full revival of multiboot >>> - >>> - there may be still controversial issues resolved before this >>> patch is ready to be merged >>> >>> >>> LESS IMPORTANT >>> >>> - [ENHANCEMENTS] Support new hypervisors >>> - >>> - qboot >>> - NEMU >>> - Hyperkit - https://github.com/cloudius-systems/osv/issues/948 >>> - >>> - multiboot - >>> https://github.com/cloudius-systems/osv/issues/981 >>> - File systems >>> - >>> - RAMFS >>> - >>> - [LESS MEMORY] Avoid creating another copy when mmap-ing RAMF >>> - https://github.com/cloudius-systems/osv/issues/979 >>> - [BUG] readdir does not handle asynchronous entry removal >>> properly - https://github.com/cloudius-systems/osv/issues/68 >>> - ZFS >>> - >>> - [ENHANCEMENTS] create ZFS image on host instead of booting >>> OSv - see https://github.com/cloudius-systems/osv/issues/918 >>> - Make VFS locking more granular >>> - >>> - VFS: read()/write() lock the vnode for too long - >>> https://github.com/cloudius-systems/osv/issues/450 >>> - >>> - this commit addresses ZFS - >>> >>> https://github.com/cloudius-systems/osv/commit/b5eadc37f12c4b97a52705830d2d9097498049c2 >>> >>> - >>> - people reported that some FS apps did not scale even >>> though commit above addresses >>> - this commit >>> >>> https://github.com/cloudius-systems/osv/commit/697943f631960b3d55f66381581e1435b726a1a8 >>> reverted >>> previous one because of >>> https://github.com/cloudius-systems/osv/issues/504 >>> - Async IO - >>> https://github.com/cloudius-systems/osv/issues/656 >>> - [MONITORING] REST API enhancements >>> - >>> - Remote test execution - >>> https://github.com/cloudius-systems/osv/issues/445 >>> - Page in/page out - >>> https://github.com/cloudius-systems/osv/issues/465 >>> - Block IO - https://github.com/cloudius-systems/osv/issues/466 >>> - Network stat - >>> https://github.com/cloudius-systems/osv/issues/467 >>> - Load average - >>> https://github.com/cloudius-systems/osv/issues/469 >>> - [SECURITY] OpenSSL upgrade >>> - Do not use setimg.py in Makefile - >>> https://github.com/cloudius-systems/osv/issues/733 >>> >>> >>> IN PROGRESS >>> >>> - Add support for Mono, C# - >>> https://github.com/cloudius-systems/osv/issues/34 >>> - >>> - added sample app >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "OSv Development" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
