The Read-Only File System (ROFS) is a simple implementation of
a file system where data from disk can only be read from and never
written to. It is simple enough for many stateless applications
deployed on OSv which only need to read code from local disk and never
write to local disk. The ROFS is inspired and shares some ideas
from the original MFS implementation by James Root from 2015.
This initial implementation of ROFS operates without cache.
This means that ROFS reads as much data from disk as requested
per uio passed in to the read function but it does not retain/cache it
for any subsequent read of the same data.
The image built with ROFS is automatically set up to mount /tmp
as ramfs filesystem to allow writing of any transient data like logs.
The layout of the data on disk is as follows:
- Super Block (512 bytes) that contains magic number and specifies meta
information including block size and location and size of tables containing
i-nodes, dentries and symbolic links
- Files data where each file is padded to 512 bytes block
- Table of directory entries referenced by index in directory i-node
(each entry holds string with direntry name and i-node number)
- Table of symlinks referenced by symlink i-node (each entry holds symbolic link
path string)
- Table of inodes where each specifies type (dir,file,symlink) and data offset
(for files it is a block on a disk, for symlinks and directories it is an
offset in one of the 2 tables above)
Besides ROFS implementation this patch also changes VFS main to automatically
mount ROFS or ZFS. It also adds number of new metrics that are captured and
output
in verbose mode.
The images with RFS can be built by specified fs=rofs option like so:
./scripts/build image=node-fs-example -j4 fs=rofs
./scripts/build image=openjdk9-java-base,java-example -j4 fs=rofs
Potential future improvements to ROFS:
- add caching layer to read ahead more data from disk in anticipation
of sequantial access
- add compression of file segments
- memory page sharing when file mmaping (implement vnop_cache)
- add LRU logic to ROFS cache
Signed-off-by: Waldemar Kozaczuk <jwkozac...@gmail.com>
---
Makefile | 4 +
fs/rofs/rofs.hh | 117 +++++++
fs/rofs/rofs_common.cc | 85 +++++
fs/rofs/rofs_vfsops.cc | 216 ++++++++++++
fs/rofs/rofs_vnops.cc | 303 +++++++++++++++++
fs/vfs/main.cc | 67 ++--
fs/vfs/vfs_conf.cc | 3 +
fs/vfs/vfs_vnode.cc | 9 +-
include/osv/vnode.h | 1 +
licenses/mfs.txt | 46 +++
loader.cc | 24 +-
modules/httpserver-api/api/fs.cc | 4 +-
scripts/build | 22 +-
scripts/gen-rofs-img.py | 373 +++++++++++++++++++++
static/etc/fstab_rofs | 4 +
usr_nozfs.manifest.skel => usr_ramfs.manifest.skel | 0
usr_nozfs.manifest.skel => usr_rofs.manifest.skel | 2 +-
17 files changed, 1245 insertions(+), 35 deletions(-)
create mode 100644 fs/rofs/rofs.hh
create mode 100644 fs/rofs/rofs_common.cc
create mode 100644 fs/rofs/rofs_vfsops.cc
create mode 100644 fs/rofs/rofs_vnops.cc
create mode 100644 licenses/mfs.txt
create mode 100755 scripts/gen-rofs-img.py
create mode 100644 static/etc/fstab_rofs
copy usr_nozfs.manifest.skel => usr_ramfs.manifest.skel (100%)
rename usr_nozfs.manifest.skel => usr_rofs.manifest.skel (88%)
diff --git a/Makefile b/Makefile
index 8dd2371..2a02a19 100644
--- a/Makefile
+++ b/Makefile
@@ -1769,6 +1769,10 @@ fs_objs += ramfs/ramfs_vfsops.o \
fs_objs += devfs/devfs_vnops.o \
devfs/device.o
+fs_objs += rofs/rofs_vfsops.o \
+ rofs/rofs_vnops.o \
+ rofs/rofs_common.o
+
fs_objs += procfs/procfs_vnops.o
objects += $(addprefix fs/, $(fs_objs))
diff --git a/fs/rofs/rofs.hh b/fs/rofs/rofs.hh
new file mode 100644
index 0000000..9fcd9a9
--- /dev/null
+++ b/fs/rofs/rofs.hh
@@ -0,0 +1,117 @@
+/*
+ * Copyright (c) 2015 Carnegie Mellon University.
+ * All Rights Reserved.
+ *
+ * THIS SOFTWARE IS PROVIDED "AS IS," WITH NO WARRANTIES WHATSOEVER. CARNEGIE
+ * MELLON UNIVERSITY EXPRESSLY DISCLAIMS TO THE FULLEST EXTENT PERMITTEDBY LAW
+ * ALL EXPRESS, IMPLIED, AND STATUTORY WARRANTIES, INCLUDING, WITHOUT
+ * LIMITATION, THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
+ * PURPOSE, AND NON-INFRINGEMENT OF PROPRIETARY RIGHTS.
+ *
+ * Released under a modified BSD license. For full terms, please see mfs.txt in
+ * the licenses folder or contact permi...@sei.cmu.edu.
+ *
+ * DM-0002621
+ *
+ * Based on https://github.com/jdroot/mfs
+ *
+ * Copyright (C) 2017 Waldemar Kozaczuk
+ * Inspired by original MFS implementation by James Root from 2015
+ *
+ * This work is open source software, licensed under the terms of the
+ * BSD license as described in the LICENSE file in the top-level directory.
+ */
+
+//
+// The Read-Only File System (ROFS) provides simple implementation of
+// a file system where data from disk can only be read from and never
+// written to. It is simple enough for many stateless applications
+// deployed on OSv which only need to read code from local disk and never
+// write to local disk. The ROFS is inspired and shares some ideas
+// from the original MFS implementation by James Root from 2015.
+//
+// This initial version of ROFS operates without cache. It reads as much data
+// from disk as requested per uio passed in to the read function and it does
+// not retain/cache it for any subsequent read of the same data.
+//
+// The structure of the data on disk is explained in scripts/gen-rofs-img.py
+
+#ifndef __INCLUDE_ROFS_H__
+#define __INCLUDE_ROFS_H__
+
+#include <osv/vnode.h>
+#include <osv/mount.h>
+#include <osv/dentry.h>
+#include <osv/prex.h>
+#include <osv/buf.h>
+
+#define ROFS_VERSION 1
+#define ROFS_MAGIC 0xDEADBEAD
+
+#define ROFS_INODE_SIZE ((uint64_t)sizeof(struct rofs_inode))
+
+#define ROFS_SUPERBLOCK_SIZE sizeof(struct rofs_super_block)
+#define ROFS_SUPERBLOCK_BLOCK 0
+
+//#define ROFS_DEBUG_ENABLED 1
+
+#if defined(ROFS_DEBUG_ENABLED)
+#define print(...) kprintf(__VA_ARGS__)
+#else
+#define print(...)
+#endif
+
+#define ROFS_DIAGNOSTICS_ENABLED 1
+
+#if defined(ROFS_DIAGNOSTICS_ENABLED)
+#define ROFS_STOPWATCH_START auto begin =
std::chrono::high_resolution_clock::now();
+#define ROFS_STOPWATCH_END(total) auto end =
std::chrono::high_resolution_clock::now(); \
+std::chrono::duration<double> sec = end - begin; \
+total += ((long)(sec.count() * 1000000));
+//TODO: Review - avoid conversions
+#else
+#define ROFS_STOPWATCH_START
+#define ROFS_STOPWATCH_END(...)
+#endif
+
+extern struct vfsops rofs_vfsops;
+extern struct vnops rofs_vnops;
+
+struct rofs_super_block {
+ uint64_t magic;
+ uint64_t version;
+ uint64_t block_size;
+ uint64_t structure_info_first_block;
+ uint64_t structure_info_blocks_count;
+ uint64_t directory_entries_count;
+ uint64_t symlinks_count;
+ uint64_t inodes_count;
+};
+
+struct rofs_inode {
+ mode_t mode;
+ uint64_t inode_no;
+ uint64_t data_offset;
+ union {
+ uint64_t file_size;
+ uint64_t dir_children_count;
+ };
+};
+
+struct rofs_dir_entry {
+ char *filename;
+ uint64_t inode_no;
+};
+
+struct rofs_info {
+ struct rofs_super_block *sb;
+ struct rofs_dir_entry *dir_entries;
+ char **symlinks;
+ struct rofs_inode *inodes;
+};
+
+int rofs_read_blocks(struct device *device, uint64_t starting_block, uint64_t
blocks_count, void *buf);
+
+void rofs_set_vnode(struct vnode *vnode, struct rofs_inode *inode);
+
+#endif
diff --git a/fs/rofs/rofs_common.cc b/fs/rofs/rofs_common.cc
new file mode 100644
index 0000000..8b04f5c
--- /dev/null
+++ b/fs/rofs/rofs_common.cc
@@ -0,0 +1,85 @@
+/*
+ * Copyright (c) 2015 Carnegie Mellon University.
+ * All Rights Reserved.
+ *
+ * THIS SOFTWARE IS PROVIDED "AS IS," WITH NO WARRANTIES WHATSOEVER. CARNEGIE
+ * MELLON UNIVERSITY EXPRESSLY DISCLAIMS TO THE FULLEST EXTENT PERMITTEDBY LAW
+ * ALL EXPRESS, IMPLIED, AND STATUTORY WARRANTIES, INCLUDING, WITHOUT
+ * LIMITATION, THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
+ * PURPOSE, AND NON-INFRINGEMENT OF PROPRIETARY RIGHTS.
+ *
+ * Released under a modified BSD license. For full terms, please see mfs.txt in
+ * the licenses folder or contact permi...@sei.cmu.edu.
+ *
+ * DM-0002621
+ *
+ * Based on https://github.com/jdroot/mfs
+ *
+ * Copyright (C) 2017 Waldemar Kozaczuk
+ * Inspired by original MFS implementation by James Root from 2015
+ *
+ * This work is open source software, licensed under the terms of the
+ * BSD license as described in the LICENSE file in the top-level directory.
+ */
+
+#include "rofs.hh"
+#include <osv/device.h>
+#include <osv/bio.h>
+
+#if defined(ROFS_DIAGNOSTICS_ENABLED)
+extern std::atomic<long> rofs_block_read_count;
+extern std::atomic<long> rofs_block_read_ms;
+#endif
+
+void rofs_set_vnode(struct vnode *vnode, struct rofs_inode *inode)
+{
+ off_t size = 0;
+ if (vnode == nullptr || inode == nullptr) {
+ return;
+ }
+
+ vnode->v_data = inode;
+ vnode->v_ino = inode->inode_no;
+
+ // Set type
+ if (S_ISDIR(inode->mode)) {
+ size = ROFS_INODE_SIZE; //TODO: Revisit
+ vnode->v_type = VDIR;
+ } else if (S_ISREG(inode->mode)) {
+ size = inode->file_size;
+ vnode->v_type = VREG;
+ } else if (S_ISLNK(inode->mode)) {
+ size = 512; // TODO: Revisit
+ vnode->v_type = VLNK;
+ }
+
+ vnode->v_mode = 0555;
+ vnode->v_size = size;
+}
+
+int
+rofs_read_blocks(struct device *device, uint64_t starting_block, uint64_t
blocks_count, void *buf)
+{
+ ROFS_STOPWATCH_START
+ struct bio *bio = alloc_bio();
+ if (!bio)
+ return ENOMEM;
+
+ bio->bio_cmd = BIO_READ;
+ bio->bio_dev = device;
+ bio->bio_data = buf;
+ bio->bio_offset = starting_block << 9;
+ bio->bio_bcount = blocks_count * BSIZE;
+
+ bio->bio_dev->driver->devops->strategy(bio);
+ int error = bio_wait(bio);
+
+ destroy_bio(bio);
+
+#if defined(ROFS_DIAGNOSTICS_ENABLED)
+ rofs_block_read_count += blocks_count;
+#endif
+ ROFS_STOPWATCH_END(rofs_block_read_ms)
+
+ return error;
+}
diff --git a/fs/rofs/rofs_vfsops.cc b/fs/rofs/rofs_vfsops.cc
new file mode 100644
index 0000000..f6d7b3b
--- /dev/null
+++ b/fs/rofs/rofs_vfsops.cc
@@ -0,0 +1,216 @@
+/*
+ * Copyright (c) 2015 Carnegie Mellon University.
+ * All Rights Reserved.
+ *
+ * THIS SOFTWARE IS PROVIDED "AS IS," WITH NO WARRANTIES WHATSOEVER. CARNEGIE
+ * MELLON UNIVERSITY EXPRESSLY DISCLAIMS TO THE FULLEST EXTENT PERMITTEDBY LAW
+ * ALL EXPRESS, IMPLIED, AND STATUTORY WARRANTIES, INCLUDING, WITHOUT
+ * LIMITATION, THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
+ * PURPOSE, AND NON-INFRINGEMENT OF PROPRIETARY RIGHTS.
+ *
+ * Released under a modified BSD license. For full terms, please see mfs.txt in
+ * the licenses folder or contact permi...@sei.cmu.edu.
+ *
+ * DM-0002621
+ *
+ * Based on https://github.com/jdroot/mfs
+ *
+ * Copyright (C) 2017 Waldemar Kozaczuk
+ * Inspired by original MFS implementation by James Root from 2015
+ *
+ * This work is open source software, licensed under the terms of the
+ * BSD license as described in the LICENSE file in the top-level directory.
+ */
+
+#include "rofs.hh"
+#include <sys/types.h>
+#include <osv/device.h>
+#include <osv/debug.h>
+#include <iomanip>
+#include <iostream>
+
+static int rofs_mount(struct mount *mp, const char *dev, int flags, const void
*data);
+static int rofs_sync(struct mount *mp);
+static int rofs_statfs(struct mount *mp, struct statfs *statp);
+static int rofs_unmount(struct mount *mp, int flags);
+
+#define rofs_vget ((vfsop_vget_t)vfs_nullop)
+
+#if defined(ROFS_DIAGNOSTICS_ENABLED)
+std::atomic<long> rofs_block_read_ms(0);
+std::atomic<long> rofs_block_read_count(0);
+#endif
+
+struct vfsops rofs_vfsops = {
+ rofs_mount, /* mount */
+ rofs_unmount, /* unmount */
+ rofs_sync, /* sync */
+ rofs_vget, /* vget */
+ rofs_statfs, /* statfs */
+ &rofs_vnops /* vnops */
+};
+
+static int
+rofs_mount(struct mount *mp, const char *dev, int flags, const void *data)
+{
+ struct device *device;
+ struct rofs_info *rofs = nullptr;
+ struct rofs_super_block *sb = nullptr;
+ int error = -1;
+
+ error = device_open(dev + 5, DO_RDWR, &device);
+ if (error) {
+ kprintf("[rofs] Error opening device!\n");
+ return error;
+ }
+
+ void *buf = malloc(BSIZE); //Just enough for single block of 512 bytes
+ error = rofs_read_blocks(device, ROFS_SUPERBLOCK_BLOCK, 1, buf);
+ if (error) {
+ kprintf("[rofs] Error reading rofs superblock\n");
+ device_close(device);
+ free(buf);
+ return error;
+ }
+
+ // We see if the file system is ROFS, if not, return error and close
everything
+ sb = (struct rofs_super_block *) buf;
+ if (sb->magic != ROFS_MAGIC) {
+ print("[rofs] Error magics do not match!\n");
+ print("[rofs] Expecting %016llX but got %016llX\n", ROFS_MAGIC,
sb->magic);
+ free(buf);
+ device_close(device);
+ return -1; // TODO: Proper error code
+ }
+
+ if (sb->version != ROFS_VERSION) {
+ kprintf("[rofs] Found rofs volume but incompatible version!\n");
+ kprintf("[rofs] Expecting %llu but found %llu\n", ROFS_VERSION,
sb->version);
+ free(buf);
+ device_close(device);
+ return -1;
+ }
+
+ print("[rofs] Got superblock version: 0x%016llX\n", sb->version);
+ print("[rofs] Got magic: 0x%016llX\n", sb->magic);
+ print("[rofs] Got block size: %d\n", sb->block_size);
+ print("[rofs] Got structure info first block: %d\n",
sb->structure_info_first_block);
+ print("[rofs] Got structure info blocks count: %d\n",
sb->structure_info_blocks_count);
+ print("[rofs] Got directory entries count: %d\n",
sb->directory_entries_count);
+ print("[rofs] Got symlinks count: %d\n", sb->symlinks_count);
+ print("[rofs] Got inode count: %d\n", sb->inodes_count);
+ //
+ // Since we have found ROFS, we can copy the superblock now
+ sb = new rofs_super_block;
+ memcpy(sb, buf, ROFS_SUPERBLOCK_SIZE);
+ free(buf);
+ //
+ // Read structure_info_blocks_count to construct array of directory
enries, symlinks and i-nodes
+ buf = malloc(BSIZE * sb->structure_info_blocks_count);
+ error = rofs_read_blocks(device, sb->structure_info_first_block,
sb->structure_info_blocks_count, buf);
+ if (error) {
+ kprintf("[rofs] Error reading rofs structure info blocks\n");
+ device_close(device);
+ free(buf);
+ return error;
+ }
+
+ rofs = new struct rofs_info;
+ rofs->sb = sb;
+ rofs->dir_entries = (struct rofs_dir_entry *) malloc(sizeof(struct
rofs_dir_entry) * sb->directory_entries_count);
+
+ void *data_ptr = buf;
+ //
+ // Read directory entries
+ for (unsigned int idx = 0; idx < sb->directory_entries_count; idx++) {
+ struct rofs_dir_entry *dir_entry = &(rofs->dir_entries[idx]);
+ dir_entry->inode_no = *((uint64_t *) data_ptr);
+ data_ptr += sizeof(uint64_t);
+
+ unsigned short *filename_size = (unsigned short *) data_ptr;
+ data_ptr += sizeof(unsigned short);
+
+ dir_entry->filename = (char *) malloc(*filename_size + 1);
+ strncpy(dir_entry->filename, (char *) data_ptr, *filename_size);
+ dir_entry->filename[*filename_size] = 0;
+ print("[rofs] i-node: %d -> directory entry: %s\n",
dir_entry->inode_no, dir_entry->filename);
+ data_ptr += *filename_size * sizeof(char);
+ }
+ //
+ // Read symbolic links
+ rofs->symlinks = (char **) malloc(sizeof(char *) * sb->symlinks_count);
+
+ for (unsigned int idx = 0; idx < sb->symlinks_count; idx++) {
+ unsigned short *symlink_path_size = (unsigned short *) data_ptr;
+ data_ptr += sizeof(unsigned short);
+
+ rofs->symlinks[idx] = (char *) malloc(*symlink_path_size + 1);
+ strncpy(rofs->symlinks[idx], (char *) data_ptr, *symlink_path_size);
+ rofs->symlinks[idx][*symlink_path_size] = 0;
+ print("[rofs] symlink: %s\n", rofs->symlinks[idx]);
+ data_ptr += *symlink_path_size * sizeof(char);
+ }
+ //
+ // Read i-nodes
+ rofs->inodes = (struct rofs_inode *) malloc(sizeof(struct rofs_inode) *
sb->inodes_count);
+ memcpy(rofs->inodes, data_ptr, sb->inodes_count * sizeof(struct
rofs_inode));
+
+ for (unsigned int idx = 0; idx < sb->inodes_count; idx++) {
+ print("[rofs] inode: %d, size: %d\n", rofs->inodes[idx].inode_no,
rofs->inodes[idx].file_size);
+ }
+
+ free(buf);
+
+ // Save a reference to our superblock
+ mp->m_data = rofs;
+ mp->m_dev = device;
+
+ rofs_set_vnode(mp->m_root->d_vnode, rofs->inodes);
+
+ print("[rofs] returning from mount\n");
+
+ return 0;
+}
+
+static int rofs_sync(struct mount *mp) {
+ return 0;
+}
+
+static int rofs_statfs(struct mount *mp, struct statfs *statp)
+{
+ struct rofs_info *rofs = (struct rofs_info *) mp->m_data;
+ struct rofs_super_block *sb = rofs->sb;
+
+ statp->f_bsize = sb->block_size;
+
+ // Total blocks
+ statp->f_blocks = sb->structure_info_blocks_count +
sb->structure_info_first_block;
+ // Read only. 0 blocks free
+ statp->f_bfree = 0;
+ statp->f_bavail = 0;
+
+ statp->f_ffree = 0;
+ statp->f_files = sb->inodes_count; //Needs to be inode count
+
+ statp->f_namelen = 0; //FIXME - unlimited ROFS_FILENAME_MAXLEN;
+
+ return 0;
+}
+
+static int
+rofs_unmount(struct mount *mp, int flags)
+{
+ struct rofs_info *rofs = (struct rofs_info *) mp->m_data;
+ struct rofs_super_block *sb = rofs->sb;
+ struct device *dev = mp->m_dev;
+
+ int error = device_close(dev);
+ delete sb;
+ delete rofs;
+
+#if defined(ROFS_DIAGNOSTICS_ENABLED)
+ debugf("ROFS: spent %.2f ms reading from disk\n", ((double)
rofs_block_read_ms.load()) / 1000);
+ debugf("ROFS: read %d 512-byte blocks from disk\n",
rofs_block_read_count.load());
+#endif
+ return error;
+}
diff --git a/fs/rofs/rofs_vnops.cc b/fs/rofs/rofs_vnops.cc
new file mode 100644
index 0000000..5d2cff0
--- /dev/null
+++ b/fs/rofs/rofs_vnops.cc
@@ -0,0 +1,303 @@
+/*
+ * Copyright (c) 2015 Carnegie Mellon University.
+ * All Rights Reserved.
+ *
+ * THIS SOFTWARE IS PROVIDED "AS IS," WITH NO WARRANTIES WHATSOEVER. CARNEGIE
+ * MELLON UNIVERSITY EXPRESSLY DISCLAIMS TO THE FULLEST EXTENT PERMITTEDBY LAW
+ * ALL EXPRESS, IMPLIED, AND STATUTORY WARRANTIES, INCLUDING, WITHOUT
+ * LIMITATION, THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
+ * PURPOSE, AND NON-INFRINGEMENT OF PROPRIETARY RIGHTS.
+ *
+ * Released under a modified BSD license. For full terms, please see mfs.txt in
+ * the licenses folder or contact permi...@sei.cmu.edu.
+ *
+ * DM-0002621
+ *
+ * Based on https://github.com/jdroot/mfs
+ *
+ * Copyright (C) 2017 Waldemar Kozaczuk
+ * Inspired by original MFS implementation by James Root from 2015
+ *
+ * This work is open source software, licensed under the terms of the
+ * BSD license as described in the LICENSE file in the top-level directory.
+ */
+
+#include <sys/stat.h>
+#include <dirent.h>
+#include <sys/param.h>
+
+#include <errno.h>
+#include <string.h>
+#include <stdlib.h>
+#include <fcntl.h>
+
+#include <osv/prex.h>
+#include <osv/vnode.h>
+#include <osv/file.h>
+#include <osv/mount.h>
+#include <osv/debug.h>
+
+#include <sys/types.h>
+#include <osv/device.h>
+#include <osv/sched.hh>
+
+#include "rofs.hh"
+
+#define VERIFY_READ_INPUT_ARGUMENTS() \
+ /* Cant read directories */\
+ if (vnode->v_type == VDIR) \
+ return EISDIR; \
+ /* Cant read anything but reg */\
+ if (vnode->v_type != VREG) \
+ return EINVAL; \
+ /* Cant start reading before the first byte */\
+ if (uio->uio_offset < 0) \
+ return EINVAL; \
+ /* Need to read more than 1 byte */\
+ if (uio->uio_resid == 0) \
+ return 0; \
+ /* Cant read after the end of the file */\
+ if (uio->uio_offset >= (off_t)vnode->v_size) \
+ return 0;
+
+int rofs_init(void) {
+ return 0;
+}
+
+static int rofs_open(struct file *fp)
+{
+ if ((file_flags(fp) & FWRITE)) {
+ // Do no allow opening files to write
+ return (EROFS);
+ }
+ print("[rofs] rofs_open called for inode [%d] \n",
+ ((struct rofs_inode *)
fp->f_dentry.get()->d_vnode->v_data)->inode_no);
+ return 0;
+}
+
+static int rofs_close(struct vnode *vp, struct file *fp) {
+ print("[rofs] rofs_close called\n");
+ // Nothing to do really...
+ return 0;
+}
+
+//
+// This function reads symbolic link information from directory structure in
memory
+// under rofs->symlinks table
+static int rofs_readlink(struct vnode *vnode, struct uio *uio)
+{
+ struct rofs_info *rofs = (struct rofs_info *) vnode->v_mount->m_data;
+ struct rofs_inode *inode = (struct rofs_inode *) vnode->v_data;
+
+ if (!S_ISLNK(inode->mode)) {
+ return EINVAL; //This node is not a symbolic link
+ }
+
+ assert(inode->data_offset >= 0 && inode->data_offset <
rofs->sb->symlinks_count);
+
+ char *link_path = rofs->symlinks[inode->data_offset];
+
+ print("[rofs] rofs_readlink returned link [%s]\n", link_path);
+ return uiomove(link_path, strlen(link_path), uio);
+}
+
+//
+// This function reads as much data as requested per uio in single read from
the disk but
+// the data does not get retained for subsequent reads
+static int rofs_read(struct vnode *vnode, struct file *fp, struct uio *uio,
int ioflag)
+{
+ struct rofs_info *rofs = (struct rofs_info *) vnode->v_mount->m_data;
+ struct rofs_super_block *sb = rofs->sb;
+ struct rofs_inode *inode = (struct rofs_inode *) vnode->v_data;
+ struct device *device = vnode->v_mount->m_dev;
+
+ VERIFY_READ_INPUT_ARGUMENTS()
+
+ int rv = 0;
+ int error = -1;
+ uint64_t block = inode->data_offset;
+ uint64_t offset = 0;
+
+ // Total read amount is what they requested, or what is left
+ uint64_t read_amt = std::min<uint64_t>(inode->file_size - uio->uio_offset,
uio->uio_resid);
+
+ // Calculate which block we need actually need to read
+ block += uio->uio_offset / sb->block_size;
+ offset = uio->uio_offset % sb->block_size;
+
+ uint64_t block_count = (offset + read_amt) / sb->block_size;
+ if ((offset + read_amt) % sb->block_size > 0)
+ block_count++;
+
+ void *buf = malloc(BSIZE * block_count);
+
+ print("[rofs] rofs_read [%d], inode: %d, [%d -> %d] at %d of %d bytes\n",
+ sched::thread::current()->id(), inode->inode_no, block, block_count,
uio->uio_offset, read_amt);
+
+ error = rofs_read_blocks(device, block, block_count, buf);
+
+ if (error) {
+ kprintf("[rofs_read] Error reading data\n");
+ free(buf);
+ return error;
+ }
+
+ rv = uiomove(buf + offset, read_amt, uio);
+
+ free(buf);
+ return rv;
+}
+
+//
+// This functions reads directory information (dentries) based on information
in memory
+// under rofs->dir_entries table
+static int rofs_readdir(struct vnode *vnode, struct file *fp, struct dirent
*dir)
+{
+ struct rofs_info *rofs = (struct rofs_info *) vnode->v_mount->m_data;
+ struct rofs_inode *inode = (struct rofs_inode *) vnode->v_data;
+
+ uint64_t index = 0;
+
+ if (!S_ISDIR(inode->mode)) {
+ return ENOTDIR;
+ }
+
+ if (fp->f_offset == 0) {
+ dir->d_type = DT_DIR;
+ strlcpy((char *) &dir->d_name, ".", sizeof(dir->d_name));
+ } else if (fp->f_offset == 1) {
+ dir->d_type = DT_DIR;
+ strlcpy((char *) &dir->d_name, "..", sizeof(dir->d_name));
+ } else {
+ index = fp->f_offset - 2;
+ if (index >= inode->dir_children_count) {
+ return ENOENT;
+ }
+
+ dir->d_fileno = fp->f_offset;
+
+ // Set the name
+ struct rofs_dir_entry *directory_entry = rofs->dir_entries +
(inode->data_offset + index);
+ strlcpy((char *) &dir->d_name, directory_entry->filename,
sizeof(dir->d_name));
+ dir->d_ino = directory_entry->inode_no;
+
+ struct rofs_inode *directory_entry_inode = rofs->inodes + (dir->d_ino
- 1);
+ if (S_ISDIR(directory_entry_inode->mode))
+ dir->d_type = DT_DIR;
+ else if (S_ISLNK(directory_entry_inode->mode))
+ dir->d_type = DT_LNK;
+ else
+ dir->d_type = DT_REG;
+ }
+
+ fp->f_offset++;
+
+ return 0;
+}
+
+//
+// This functions looks up directory entry based on the directory information
stored in memory
+// under rofs->dir_entries table
+static int rofs_lookup(struct vnode *vnode, char *name, struct vnode **vpp)
+{
+ struct rofs_info *rofs = (struct rofs_info *) vnode->v_mount->m_data;
+ struct rofs_inode *inode = (struct rofs_inode *) vnode->v_data;
+ struct vnode *vp = nullptr;
+
+ if (*name == '\0') {
+ return ENOENT;
+ }
+
+ if (!S_ISDIR(inode->mode)) {
+ print("[rofs] ABORTED lookup up %s at inode %d because not a
directory\n", name, inode->inode_no);
+ return ENOTDIR;
+ }
+
+ for (unsigned int idx = 0; idx < inode->dir_children_count; idx++) {
+ if (strcmp(name, rofs->dir_entries[inode->data_offset + idx].filename)
== 0) {
+ int inode_no = rofs->dir_entries[inode->data_offset +
idx].inode_no;
+
+ if (vget(vnode->v_mount, inode_no, &vp)) { //TODO: Will it ever
work? Revisit
+ print("[rofs] found vp in cache!\n");
+ *vpp = vp;
+ return 0;
+ }
+
+ struct rofs_inode *found_inode = rofs->inodes + (inode_no - 1);
//Check if exists
+ rofs_set_vnode(vp, found_inode);
+
+ print("[rofs] found the directory entry [%s] at at inode %d ->
%d!\n", name, inode->inode_no,
+ found_inode->inode_no);
+
+ *vpp = vp;
+ return 0;
+ }
+ }
+
+ print("[rofs] FAILED to find up %s\n", name);
+
+ return ENOENT;
+}
+
+static int rofs_getattr(struct vnode *vnode, struct vattr *attr)
+{
+ struct rofs_inode *inode = (struct rofs_inode *) vnode->v_data;
+
+ attr->va_mode = 0555;
+
+ if (S_ISDIR(inode->mode)) {
+ attr->va_type = VDIR;
+ } else if (S_ISREG(inode->mode)) {
+ attr->va_type = VREG;
+ } else if (S_ISLNK(inode->mode)) {
+ attr->va_type = VLNK;
+ }
+
+ attr->va_nodeid = vnode->v_ino;
+ attr->va_size = vnode->v_size;
+
+ return 0;
+}
+
+#define rofs_write ((vnop_write_t)vop_erofs)
+#define rofs_seek ((vnop_seek_t)vop_nullop)
+#define rofs_ioctl ((vnop_ioctl_t)vop_nullop)
+#define rofs_create ((vnop_create_t)vop_erofs)
+#define rofs_remove ((vnop_remove_t)vop_erofs)
+#define rofs_rename ((vnop_rename_t)vop_erofs)
+#define rofs_mkdir ((vnop_mkdir_t)vop_erofs)
+#define rofs_rmdir ((vnop_rmdir_t)vop_erofs)
+#define rofs_setattr ((vnop_setattr_t)vop_erofs)
+#define rofs_inactive ((vnop_inactive_t)vop_nullop)
+#define rofs_truncate ((vnop_truncate_t)vop_erofs)
+#define rofs_link ((vnop_link_t)vop_erofs)
+#define rofs_arc ((vnop_cache_t) nullptr)
+#define rofs_fallocate ((vnop_fallocate_t)vop_erofs)
+#define rofs_fsync ((vnop_fsync_t)vop_nullop)
+#define rofs_symlink ((vnop_symlink_t)vop_erofs)
+
+struct vnops rofs_vnops = {
+ rofs_open, /* open */
+ rofs_close, /* close */
+ rofs_read, /* read */
+ rofs_write, /* write - returns error when called */
+ rofs_seek, /* seek */
+ rofs_ioctl, /* ioctl */
+ rofs_fsync, /* fsync */
+ rofs_readdir, /* readdir */
+ rofs_lookup, /* lookup */
+ rofs_create, /* create - returns error when called */
+ rofs_remove, /* remove - returns error when called */
+ rofs_rename, /* rename - returns error when called */
+ rofs_mkdir, /* mkdir - returns error when called */
+ rofs_rmdir, /* rmdir - returns error when called */
+ rofs_getattr, /* getattr */
+ rofs_setattr, /* setattr - returns error when called */
+ rofs_inactive, /* inactive */
+ rofs_truncate, /* truncate - returns error when called*/
+ rofs_link, /* link - returns error when called*/
+ rofs_arc, /* arc */ //TODO: Implement to allow memory
re-use when mapping files
+ rofs_fallocate, /* fallocate - returns error when called*/
+ rofs_readlink, /* read link */
+ rofs_symlink /* symbolic link - returns error when called*/
+};
diff --git a/fs/vfs/main.cc b/fs/vfs/main.cc
index e053a78..cd619f5 100644
--- a/fs/vfs/main.cc
+++ b/fs/vfs/main.cc
@@ -2249,26 +2249,9 @@ static void import_extra_zfs_pools(void)
}
}
-extern "C" void mount_zfs_rootfs(bool pivot_root)
+void pivot_rootfs(const char* path)
{
- int ret;
-
- if (mkdir("/zfs", 0755) < 0)
- kprintf("failed to create /zfs, error = %s\n", strerror(errno));
-
- ret = sys_umount("/dev");
- if (ret)
- kprintf("failed to unmount /dev, error = %s\n", strerror(ret));
-
- ret = sys_mount("/dev/vblk0.1", "/zfs", "zfs", 0, (void *)"osv/zfs");
- if (ret)
- kprintf("failed to mount /zfs, error = %s\n", strerror(ret));
-
- if (!pivot_root) {
- return;
- }
-
- ret = sys_pivot_root("/zfs", "/");
+ int ret = sys_pivot_root(path, "/");
if (ret)
kprintf("failed to pivot root, error = %s\n", strerror(ret));
@@ -2294,6 +2277,52 @@ extern "C" void mount_zfs_rootfs(bool pivot_root)
}
}
endmntent(ent);
+}
+
+extern "C" void unmount_devfs()
+{
+ int ret = sys_umount("/dev");
+ if (ret)
+ kprintf("failed to unmount /dev, error = %s\n", strerror(ret));
+}
+
+extern "C" int mount_rofs_rootfs(bool pivot_root)
+{
+ int ret;
+
+ if (mkdir("/rofs", 0755) < 0)
+ kprintf("failed to create /rofs, error = %s\n", strerror(errno));
+
+ ret = sys_mount("/dev/vblk0.1", "/rofs", "rofs", MNT_RDONLY, 0);
+
+ if (ret) {
+ kprintf("failed to mount /rofs, error = %s\n", strerror(ret));
+ rmdir("/rofs");
+ return ret;
+ }
+
+ if (pivot_root) {
+ pivot_rootfs("/rofs");
+ }
+
+ return 0;
+}
+
+extern "C" void mount_zfs_rootfs(bool pivot_root)
+{
+ if (mkdir("/zfs", 0755) < 0)
+ kprintf("failed to create /zfs, error = %s\n", strerror(errno));
+
+ int ret = sys_mount("/dev/vblk0.1", "/zfs", "zfs", 0, (void *)"osv/zfs");
+
+ if (ret)
+ kprintf("failed to mount /zfs, error = %s\n", strerror(ret));
+
+ if (!pivot_root) {
+ return;
+ }
+
+ pivot_rootfs("/zfs");
import_extra_zfs_pools();
}
diff --git a/fs/vfs/vfs_conf.cc b/fs/vfs/vfs_conf.cc
index 59e1b79..4ac695b 100644
--- a/fs/vfs/vfs_conf.cc
+++ b/fs/vfs/vfs_conf.cc
@@ -46,12 +46,14 @@
#include "vfs.h"
extern struct vfsops ramfs_vfsops;
+extern struct vfsops rofs_vfsops;
extern struct vfsops devfs_vfsops;
extern struct vfsops nfs_vfsops;
extern struct vfsops procfs_vfsops;
extern struct vfsops zfs_vfsops;
extern int ramfs_init(void);
+extern int rofs_init(void);
extern int devfs_init(void);
extern int nfs_init(void);
extern int procfs_init(void);
@@ -66,5 +68,6 @@ const struct vfssw vfssw[] = {
{"nfs", nfs_init, &nfs_vfsops},
{"procfs", procfs_init, &procfs_vfsops},
{"zfs", zfs_init, &zfs_vfsops},
+ {"rofs", rofs_init, &rofs_vfsops},
{nullptr, fs_noop, nullptr},
};
diff --git a/fs/vfs/vfs_vnode.cc b/fs/vfs/vfs_vnode.cc
index 7e1c7ab..13d1e5c 100644
--- a/fs/vfs/vfs_vnode.cc
+++ b/fs/vfs/vfs_vnode.cc
@@ -478,24 +478,27 @@ vnode_dump(void)
int
vop_nullop(void)
{
-
return 0;
}
int
vop_einval(void)
{
-
return EINVAL;
}
int
vop_eperm(void)
{
-
return EPERM;
}
+int
+vop_erofs(void)
+{
+ return EROFS;
+}
+
/*
* vnode_init() is called once (from vfs_init)
* in initialization.
diff --git a/include/osv/vnode.h b/include/osv/vnode.h
index 45bd93e..e35aa83 100755
--- a/include/osv/vnode.h
+++ b/include/osv/vnode.h
@@ -210,6 +210,7 @@ struct vnops {
int vop_nullop(void);
int vop_einval(void);
int vop_eperm(void);
+int vop_erofs(void);
struct vnode *vn_lookup(struct mount *, uint64_t);
void vn_lock(struct vnode *);
void vn_unlock(struct vnode *);
diff --git a/licenses/mfs.txt b/licenses/mfs.txt
new file mode 100644
index 0000000..b7061e0
--- /dev/null
+++ b/licenses/mfs.txt
@@ -0,0 +1,46 @@
+Copyright (c) 2015 Carnegie Mellon University.
+All Rights Reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+ 1. Redistributions of source code must retain the above copyright notice,
+ this list of conditions and the following acknowledgments and
+ disclaimers.
+
+ 2. Redistributions in binary form must reproduce the above copyright
+ notice, this list of conditions and the following acknowledgments and
+ disclaimers in the documentation and/or other materials provided with
+ the distribution.
+
+ 3. Products derived from this software may not include “Carnegie Mellon
+ University,” "SEI” and/or “Software Engineering Institute" in the name
+ of such derived product, nor shall “Carnegie Mellon University,” "SEI”
+ and/or “Software Engineering Institute" be used to endorse or promote
+ products derived from this software without prior written permission.
+ For written permission, please contact permi...@sei.cmu.edu.
+
+ACKNOWLEDGMENTS AND DISCLAIMERS:
+Copyright 2015 Carnegie Mellon University
+
+This material is based upon work funded and supported by the Department of
+Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University
+for the operation of the Software Engineering Institute, a federally funded
+research and development center.
+
+Any opinions, findings and conclusions or recommendations expressed in this
+material are those of the author(s) and do not necessarily reflect the views of
+the United States Department of Defense.
+
+NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE
+MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO
+WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER
+INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR
+MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL.
+CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT
+TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.
+
+This material has been approved for public release and unlimited distribution.
+
+DM-0002621
+
diff --git a/loader.cc b/loader.cc
index c3a7256..1b2892c 100644
--- a/loader.cc
+++ b/loader.cc
@@ -84,7 +84,10 @@ void setup_tls(elf::init_table inittab)
extern "C" {
void premain();
void vfs_init(void);
+ void unmount_devfs();
void mount_zfs_rootfs(bool);
+ int mount_rofs_rootfs(bool);
+ void rofs_disable_cache();
}
void premain()
@@ -165,7 +168,7 @@ void parse_options(int loader_argc, char** loader_argv)
("maxnic", bpo::value<int>(), "maximum NIC number")
("norandom", "don't initialize any random device")
("noshutdown", "continue running after main() returns")
- ("power-off-on-abort", "use poweroff instead of halt if it's aborted")
+ ("power-off-on-abort", "use poweroff instead of halt if it's aborted")
("noinit", "don't run commands from /init")
("verbose", "be verbose, print debug messages")
("console", bpo::value<std::vector<std::string>>(), "select console
driver")
@@ -335,11 +338,22 @@ void* do_main_thread(void *_main_args)
boot_time.event("drivers loaded");
if (opt_mount) {
- zfsdev::zfsdev_init();
- mount_zfs_rootfs(opt_pivot);
- bsd_shrinker_init();
+ unmount_devfs();
+ //
+ // Try to mount rofs
+ if(mount_rofs_rootfs(opt_pivot) != 0) {
+ //
+ // Failed -> try to mount zfs
+ zfsdev::zfsdev_init();
+ mount_zfs_rootfs(opt_pivot);
+ bsd_shrinker_init();
+
+ boot_time.event("ZFS mounted");
+ }
+ else {
+ boot_time.event("ROFS mounted");
+ }
}
- boot_time.event("ZFS mounted");
bool has_if = false;
osv::for_each_if([&has_if] (std::string if_name) {
diff --git a/modules/httpserver-api/api/fs.cc b/modules/httpserver-api/api/fs.cc
index e30c840..4151cd7 100644
--- a/modules/httpserver-api/api/fs.cc
+++ b/modules/httpserver-api/api/fs.cc
@@ -51,7 +51,7 @@ void init(routes& routes) {
vector<httpserver::json::DFStat> dfstats;
for (mount_desc mount : osv::current_mounts()) {
- if (mount.type == "zfs" && (onemount == "" || onemount ==
mount.path)) {
+ if ((mount.type == "zfs" || mount.type == "rofs") && (onemount ==
"" || onemount == mount.path)) {
if (statvfs(mount.path.c_str(),&st) != 0) {
throw not_found_exception("mount does not exist");
}
@@ -75,7 +75,7 @@ void init(routes& routes) {
vector<httpserver::json::DFStat> res;
for (osv::mount_desc mount : osv::current_mounts()) {
- if (mount.type == "zfs") {
+ if (mount.type == "zfs" || mount.type == "rofs") {
if (statvfs(mount.path.c_str(),&st) == 0) {
fill_dfstat(dfstat, mount, st);
res.push_back(dfstat);
diff --git a/scripts/build b/scripts/build
index f7cd0ee..eac860f 100755
--- a/scripts/build
+++ b/scripts/build
@@ -134,9 +134,10 @@ manifest=$OUT/bootfs.manifest
fs_type=${vars[fs]-zfs}
usrskel_arg=
case $fs_type in
-zfs) ;; # Nothing to change here. This is our default behavior
+zfs);; # Nothing to change here. This is our default behavior
+rofs) usrskel_arg="--usrskel usr_rofs.manifest.skel";;
ramfs) manifest=$OUT/usr.manifest
- usrskel_arg="--usrskel usr_nozfs.manifest.skel";;
+ usrskel_arg="--usrskel usr_ramfs.manifest.skel";;
*) echo "Unknown filesystem \"$fs_type\"" >&2
exit 2
esac
@@ -170,8 +171,7 @@ then
fi
loader_size=`stat --printf %s $OUT/loader.img`
-zfs_start=$(($loader_size+2097151 & ~2097151))
-zfs_size=$(($fs_size - $zfs_start))
+kernel_end=$(($loader_size+2097151 & ~2097151))
# The python scripts called below assume the current directory is $OUT (as was
# the case in our old build.mk).
@@ -180,7 +180,8 @@ cd $OUT
case $fs_type in
zfs)
cp loader.img bare.raw
- $SRC/scripts/imgedit.py setpartition "-f raw bare.raw" 2 $zfs_start
$zfs_size
+ zfs_size=$(($fs_size - $kernel_end))
+ $SRC/scripts/imgedit.py setpartition "-f raw bare.raw" 2 $kernel_end
$zfs_size
qemu-img convert -f raw -O qcow2 bare.raw usr.img
qemu-img resize usr.img ${fs_size}b >/dev/null 2>&1
@@ -193,6 +194,17 @@ zfs)
$SRC/scripts/export_manifest.py -e "$export_dir" -m
usr.manifest -D jdkbase=$jdkbase -D gccbase=$gccbase -D glibcbase=$glibcbase -D
miscbase=$miscbase
fi
;;
+rofs)
+ rm -rf rofs.img
+ $SRC/scripts/gen-${fs_type}-img.py -o rofs.img -m usr.manifest -D
jdkbase=$jdkbase -D gccbase=$gccbase -D glibcbase=$glibcbase -D
miscbase=$miscbase
+ rofs_size=`stat --printf %s rofs.img`
+ img_size=$((kernel_end + rofs_size))
+ cp loader.img bare.raw
+ $SRC/scripts/imgedit.py setpartition "-f raw bare.raw" 2 $kernel_end
$rofs_size
+ qemu-img resize bare.raw ${img_size}b >/dev/null 2>&1
+ dd if=rofs.img of=bare.raw obs=${kernel_end} seek=1 >/dev/null 2>&1
+ qemu-img convert -f raw -O qcow2 bare.raw usr.img
+ ;;
ramfs)
qemu-img convert -f raw -O qcow2 loader.img usr.img
;;
diff --git a/scripts/gen-rofs-img.py b/scripts/gen-rofs-img.py
new file mode 100755
index 0000000..e825c63
--- /dev/null
+++ b/scripts/gen-rofs-img.py
@@ -0,0 +1,373 @@
+#!/usr/bin/python
+
+#
+# Copyright (c) 2015 Carnegie Mellon University.
+# All Rights Reserved.
+#
+# THIS SOFTWARE IS PROVIDED "AS IS," WITH NO WARRANTIES WHATSOEVER. CARNEGIE
+# MELLON UNIVERSITY EXPRESSLY DISCLAIMS TO THE FULLEST EXTENT PERMITTEDBY LAW
+# ALL EXPRESS, IMPLIED, AND STATUTORY WARRANTIES, INCLUDING, WITHOUT
+# LIMITATION, THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
+# PURPOSE, AND NON-INFRINGEMENT OF PROPRIETARY RIGHTS.
+#
+# Released under a modified BSD license. For full terms, please see mfs.txt in
+# the licenses folder or contact permi...@sei.cmu.edu.
+#
+# DM-0002621
+#
+#
+# Copyright (C) 2017 Waldemar Kozaczuk
+# Inspired by original MFS implementation by James Root from 2015
+#
+# This work is open source software, licensed under the terms of the
+# BSD license as described in the LICENSE file in the top-level directory.
+#
+
+##################################################################################
+# The layout of data on the disk in the block order:
+#
+# Super Block (512 bytes) that contains magic number and specifies meta
+# information including block size and location and size of tables containing
+# i-nodes, dentries and symbolic links
+#
+# Files data where each file is padded to 512 bytes block
+#
+# Table of directory entries referenced by index in directory i-node
+# (each entry holds string with direntry name and i-node number)
+#
+# Table of symlinks referenced by symlink i-node (each entry holds symbolic
link
+# path string)
+#
+# Table of inodes where each specifies type (dir,file,symlink) and data offset
+# (for files it is a block on a disk, for symlinks and directories it is an
+# offset in one of the 2 tables above)
+##################################################################################
+
+import os, optparse, io
+from struct import *
+from ctypes import *
+from manifest_common import add_var, expand, unsymlink, read_manifest,
defines, strip_file
+
+OSV_BLOCK_SIZE = 512
+
+DIR_MODE = int('0x4000', 16)
+REG_MODE = int('0x8000', 16)
+LINK_MODE = int('0xA000', 16)
+
+block = 0
+
+class SuperBlock(Structure):
+ _fields_ = [
+ ('magic', c_ulonglong),
+ ('version', c_ulonglong),
+ ('block_size', c_ulonglong),
+ ('structure_info_first_block', c_ulonglong),
+ ('structure_info_blocks_count', c_ulonglong),
+ ('directory_entries_count', c_ulonglong),
+ ('symlinks_count', c_ulonglong),
+ ('inodes_count', c_ulonglong)
+ ]
+
+# data_offset and count represent different things depending on mode:
+# file - number of first block on disk and size in bytes (number of blocks can
be deduced)
+# directory - index of the first entry in the directory entries array and
number of entries
+# symlink - index of the entry in the symlink entries array and 1
+class Inode(Structure):
+ _fields_ = [
+ ('mode', c_ulonglong),
+ ('inode_no', c_ulonglong), #redundant
+ ('data_offset', c_ulonglong),
+ ('count', c_ulonglong) # either file size or children count
+ ]
+
+# Represents directory entry - file, subdirectory or symlink
+# It has a name and i-node number so to know what type of
+# entry it is one has to read the i-node
+# filename (length: unsigned short followed by characters)
+class DirectoryEntry(object):
+ def __init__(self,filename,inode_no):
+ self.filename = filename
+ self.inode_no = inode_no
+
+ def write(self,fp):
+ pos = fp.tell()
+ fp.write(c_ulonglong(self.inode_no))
+ fp.write(c_ushort(len(self.filename)))
+ for c in self.filename:
+ fp.write(c_char(c))
+
+ return fp.tell() - pos
+
+class SymbolicLink(object):
+ def __init__(self,path):
+ self.path = path
+
+ def write(self,fp):
+ pos = fp.tell()
+ fp.write(c_ushort(len(self.path)))
+ for c in self.path:
+ fp.write(c_char(c))
+
+ return fp.tell() - pos
+
+directory_entries = []
+directory_entries_count = 0
+
+symlinks = []
+symlinks_count = 0
+
+inodes = []
+inodes_count = 1
+
+def next_directory_entry(filename,inode_no):
+ global directory_entries
+ global directory_entries_count
+
+ directory_entry = DirectoryEntry(filename,inode_no)
+ directory_entries_count += 1
+ directory_entries.append(directory_entry)
+
+ return directory_entry
+
+def next_symlink(path,directory):
+ global symlinks
+ global symlinks_count
+
+ symlink = SymbolicLink(path)
+ symlinks_count += 1
+ symlinks.append(symlink)
+
+ return symlink
+
+def next_inode():
+ global inodes_count
+ global inodes
+
+ inode = Inode()
+ inode.inode_no = inodes_count
+ inodes_count += 1
+ inodes.append(inode)
+
+ return inode
+
+def pad(fp, size):
+ fp.write('\0' * size)
+ return size
+
+def write_initial_superblock(fp):
+ global block
+ pad(fp, OSV_BLOCK_SIZE) # superblock is empty at first
+ block += 1
+
+def write_file(fp, path):
+ global block
+
+ total = 0
+ last = 0
+
+ with open(path, 'rb') as f:
+ while True:
+ chunk = f.read(OSV_BLOCK_SIZE)
+ if chunk:
+ last = len(chunk)
+ total += last
+ block += 1
+ fp.write(chunk)
+ else:
+ break
+
+ if total > 0:
+ pad(fp, OSV_BLOCK_SIZE - last)
+
+ return total
+
+def write_inodes(fp):
+ global inodes
+
+ for inode in inodes:
+ fp.write(inode)
+
+ return len(inodes) * sizeof(Inode)
+
+def write_array(fp, vals):
+ bytes_written = 0
+
+ for val in vals:
+ bytes_written += val.write(fp)
+
+ return bytes_written
+
+def write_dir(fp, manifest, dirpath, parent_dir):
+ global directory_entries_count
+
+ directory_entry_inodes = []
+ for entry in manifest:
+ inode = next_inode()
+ directory_entry_inodes.append((entry,inode))
+
+ val = manifest.get(entry)
+ if type(val) is dict: # directory
+ inode.mode = DIR_MODE
+ count, directory_entries_index = write_dir(fp, val, dirpath + '/'
+ entry, manifest)
+ inode.count = count
+ inode.data_offset = directory_entries_index
+ else: # file or symlink
+ if val.startswith('->'): #symlink
+ inode.mode = LINK_MODE
+ global symlinks_count
+ inode.data_offset = symlinks_count
+ inode.count = 1
+ next_symlink(val[2:],manifest)
+ print 'Link %s to %s' % (dirpath + '/' + entry, val[2:])
+ else: #file
+ inode.mode = REG_MODE
+ global block
+ inode.data_offset = block
+ inode.count = write_file(fp, val)
+ print 'Adding %s' % (dirpath + '/' + entry)
+
+ # This needs to be added so that later we can walk the tree
+ # when fining symlinks
+ manifest['.'] = manifest
+ manifest['..'] = parent_dir
+
+ this_directory_entries_index = directory_entries_count
+ for directory_entry_inode in directory_entry_inodes:
+
next_directory_entry(directory_entry_inode[0],directory_entry_inode[1].inode_no)
+
+ this_directory_entries_count = len(directory_entry_inodes)
+ return (this_directory_entries_count, this_directory_entries_index)
+
+def write_fs(fp, manifest):
+ global block
+ global inodes
+ global directory_entries
+ global symlinks
+
+ root_inode = next_inode()
+ root_inode.mode = DIR_MODE
+
+ count, directory_entries_index = write_dir(fp, manifest.get(''), '',
manifest)
+ root_inode.count = count
+ root_inode.data_offset = directory_entries_index
+
+ block_no = block
+
+ # Write directories entries array
+ bytes_written = write_array(fp,directory_entries)
+ bytes_written += write_array(fp,symlinks)
+
+ # Write inodes!
+ write_inodes(fp)
+ bytes_written += len(inodes) * sizeof(Inode)
+
+ return (block_no, bytes_written)
+
+def gen_image(out, manifest):
+ print 'Writing image'
+ fp = open(out, 'wb')
+
+ # write the initial superblock
+ write_initial_superblock(fp)
+
+ system_structure_block, bytes_written = write_fs(fp, manifest)
+ structure_info_last_block_bytes = bytes_written % OSV_BLOCK_SIZE
+ structure_info_blocks_count = bytes_written / OSV_BLOCK_SIZE + (1 if
structure_info_last_block_bytes > 0 else 0)
+
+ pad(fp,OSV_BLOCK_SIZE - structure_info_last_block_bytes)
+
+ global inodes
+ global directory_entries
+ global symlinks
+
+ sb = SuperBlock()
+ sb.version = 1
+ sb.magic = int('0xDEADBEAD', 16)
+ sb.block_size = OSV_BLOCK_SIZE
+ sb.structure_info_first_block = system_structure_block
+ sb.structure_info_blocks_count = structure_info_blocks_count
+ sb.directory_entries_count = len(directory_entries)
+ sb.symlinks_count = len(symlinks)
+ sb.inodes_count = len(inodes)
+
+ print 'First block: %d, blocks count: %d' %
(sb.structure_info_first_block, sb.structure_info_blocks_count)
+ print 'Directory entries count %d' % sb.directory_entries_count
+ print 'Symlinks count %d' % sb.symlinks_count
+ print 'Inodes count %d' % sb.inodes_count
+
+ fp.seek(0)
+ fp.write(sb)
+
+ fp.close()
+
+def parse_manifest(manifest):
+ manifest = [(x, y % defines) for (x, y) in manifest]
+ files = list(expand(manifest))
+ files = [(x, unsymlink(y)) for (x, y) in files]
+
+ file_dict = {}
+
+ def populate_with_directory_path(path,directory):
+ tokens = path.rstrip('/').split('/')
+ dictionary = directory
+ for token in tokens[:-1]:
+ dictionary = dictionary.setdefault(token, {})
+ return (dictionary,tokens)
+
+ for name, hostname in files:
+ p = file_dict
+ if hostname.startswith('->'):
+ p, tokens = populate_with_directory_path(name,p)
+ entry = tokens[len(tokens)-1]
+ p[entry] = hostname
+ else:
+ if hostname.endswith('-stripped.so'):
+ continue
+ hostname = strip_file(hostname)
+ if os.path.islink(hostname):
+ p, tokens = populate_with_directory_path(name,p)
+ entry = tokens[len(tokens)-1]
+ link = os.readlink(hostname)
+ p[entry] = '->%s' % link
+ elif os.path.isdir(hostname):
+ for token in name.rstrip('/').split('/'):
+ p = p.setdefault(token, {})
+ else:
+ p, tokens = populate_with_directory_path(name,p)
+ entry = tokens[len(tokens)-1]
+ p[entry] = hostname
+
+ return file_dict
+
+def main():
+ make_option = optparse.make_option
+
+ opt = optparse.OptionParser(option_list=[
+ make_option('-o',
+ dest='output',
+ help='write to FILE',
+ metavar='FILE'),
+ make_option('-m',
+ dest='manifest',
+ help='read manifest from FILE',
+ metavar='FILE'),
+ make_option('-D',
+ type='string',
+ help='define VAR=DATA',
+ metavar='VAR=DATA',
+ action='callback',
+ callback=add_var),
+ ])
+
+ (options, args) = opt.parse_args()
+
+ manifest = read_manifest(options.manifest)
+
+ outfile = os.path.abspath(options.output)
+
+ manifest = parse_manifest(manifest)
+
+ gen_image(outfile, manifest)
+
+if __name__ == '__main__':
+ main()
diff --git a/static/etc/fstab_rofs b/static/etc/fstab_rofs
new file mode 100644
index 0000000..c7e3cfd
--- /dev/null
+++ b/static/etc/fstab_rofs
@@ -0,0 +1,4 @@
+/dev/vblk0.1 / rofs defaults 0 0
+none /dev devfs defaults 0 0
+none /proc procfs defaults 0 0
+none /tmp ramfs defaults 0 0
diff --git a/usr_nozfs.manifest.skel b/usr_ramfs.manifest.skel
similarity index 100%
copy from usr_nozfs.manifest.skel
copy to usr_ramfs.manifest.skel
diff --git a/usr_nozfs.manifest.skel b/usr_rofs.manifest.skel
similarity index 88%
rename from usr_nozfs.manifest.skel
rename to usr_rofs.manifest.skel
index 235b957..0db6098 100644
--- a/usr_nozfs.manifest.skel
+++ b/usr_rofs.manifest.skel
@@ -6,7 +6,7 @@
/&/etc/hosts: ../../static/&
/etc/mnttab: ->/proc/mounts
-/&/etc/fstab: ../../static/&
+/etc/fstab: ../../static/etc/fstab_rofs
/dev: ../../static
/proc: ../../static
/tmp: ../../static
--
2.7.4
--
You received this message because you are subscribed to the Google Groups "OSv
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
