[RFC] e2fsprogs: Add ext4migrate
This is work in progress The below patches add ext4migrate utility that helps in migrating a ext3 block mapped inode to ext4 extent mapped inode. I have split the patches into two. They don't build individually. The purpose of splitting the patches into two is to make the review easeir. The contents of [PATCH 2/2] is derived out of kernel source. ext4migrate command takes the below syntax ext4migrate [--display | --migrate ] image_name filename The --display option helps in displaying the block map details for an ext3 inode and extent map details for an ext4 inode. The --migrate option convert a block mapped ext3 inode to extent mapped ext4 inode. This needs to be run on an unmounted file system (offline migration). The extent insert code is derived out of the latest ext4 kernel source. I have tried to keep the code as close as possible to the kernel sources. This makes sure that any fixes for the tree building code in kernel should be easily applied to ext4migrate. The ext3_ext naming convention instead of ext4_ext found in kernel is to make sure we are in sync with rest of e2fsprogs source. The inode modification is done only at the last stage. This is to make sure that if we fail at any intermediate stage, we exit without touching the disk. Another option I am thinking is to copy the old mapping to a temporary inode. This makes sure that if by any chance the migration fails partially writing to the disk one can retrieve the file using temporary inode. The other alternative i am working on right now is to look at the online defragmentation and see if that can be adapted to achieve online migration. The patch applies on top of e2fsprogs found at http://www.kernel.org/pub/linux/kernel/people/tytso/e2fsprogs-interim/e2fsprogs-1.39-tyt1/e2fsprogs-1.39-tyt1.tar.bz2 - To unsubscribe from this list: send the line unsubscribe linux-ext4 in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/2] e2fsprogs: Add ext4migrate
From: Aneesh Kumar K.V [EMAIL PROTECTED]
Add ext4migrate utility that helps in migrating a ext3 block mapped
inode to ext4 extent mapped inode.
ext4migrate command takes the below syntax
ext4migrate [--display | --migrate ] image_name filename
The --display option helps in displaying the block map details for an ext3 inode
and extent map details for an ext4 inode. The --migrate option convert a block
mapped
ext3 inode to extent mapped ext4 inode.
This needs to be run on an unmounted file system (offline migration).
The inode modification is done only at the last stage. This is to make sure
that if we
fail at any intermediate stage, we exit without touching the disk. Another
option I am thinking
is to copy the old mapping to a temporary inode. This makes sure that if by any
chance the migration
fails partially writing to the disk one can retrieve the file using temporary
inode.
Signed-off-by: Aneesh Kumar K.V [EMAIL PROTECTED]
---
Makefile.in |3 +-
configure.in| 21 ++-
ext4migrate/Makefile.in | 66 ++
ext4migrate/migrate.c | 568 +++
ext4migrate/migrate.h | 14 ++
5 files changed, 670 insertions(+), 2 deletions(-)
create mode 100644 ext4migrate/Makefile.in
create mode 100644 ext4migrate/migrate.c
create mode 100644 ext4migrate/migrate.h
diff --git a/Makefile.in b/Makefile.in
index 0d31caa..9d8d291 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -9,9 +9,10 @@ INSTALL = @INSTALL@
@[EMAIL PROTECTED] resize
@[EMAIL PROTECTED] debugfs
[EMAIL PROTECTED]@EXT4MIGRATE_DIR= ext4migrate
LIB_SUBDIRS=lib/et lib/ss lib/e2p lib/ext2fs lib/uuid lib/blkid intl
-PROG_SUBDIRS=e2fsck $(DEBUGFS_DIR) misc $(RESIZE_DIR) tests/progs po
+PROG_SUBDIRS=e2fsck $(DEBUGFS_DIR) misc $(RESIZE_DIR) tests/progs po
$(EXT4MIGRATE_DIR)
SUBDIRS=util $(LIB_SUBDIRS) $(PROG_SUBDIRS) tests
SUBS= lib/ext2fs/ext2_types.h lib/blkid/blkid_types.h lib/uuid/uuid_types.h
diff --git a/configure.in b/configure.in
index 44b718d..5ae5e0b 100644
--- a/configure.in
+++ b/configure.in
@@ -436,6 +436,24 @@ echo Building e2fsck statically by default
)
AC_SUBST(E2FSCK_TYPE)
dnl
+dnl handle --enable-ext4migrate
+dnl
+AC_ARG_ENABLE([ext4migrate],
+[ --disable-ext4migrate disable support of ext4migrate program],
+if test $enableval = no
+then
+ echo Disabling ext4migrate support
+ EXT4MIGRATE_CMT=#
+else
+ EXT4MIGRATE_CMT=
+ echo Enabling ext4migrate support
+fi
+,
+echo Enabling ext4migrate support by default
+EXT4MIGRATE_CMT=
+)
+AC_SUBST(EXT4MIGRATE_CMT)
+dnl
dnl See whether to install the `fsck' wrapper program (that calls e2fsck)
dnl
AC_ARG_ENABLE([fsck],
@@ -862,7 +880,8 @@ for i in MCONFIG Makefile e2fsprogs.spec \
lib/e2p/e2p.pc lib/blkid/blkid.pc lib/ext2fs/ext2fs.pc \
misc/Makefile ext2ed/Makefile e2fsck/Makefile \
debugfs/Makefile tests/Makefile tests/progs/Makefile \
- resize/Makefile doc/Makefile intl/Makefile po/Makefile.in ; do
+ resize/Makefile doc/Makefile intl/Makefile po/Makefile.in \
+ ext4migrate/Makefile ; do
if test -d `dirname ${srcdir}/$i` ; then
outlist=$outlist $i
fi
diff --git a/ext4migrate/Makefile.in b/ext4migrate/Makefile.in
new file mode 100644
index 000..05b089e
--- /dev/null
+++ b/ext4migrate/Makefile.in
@@ -0,0 +1,66 @@
+#
+# Standard e2fsprogs prologue
+#
+
+srcdir = @srcdir@
+top_srcdir = @top_srcdir@
+VPATH = @srcdir@
+top_builddir = ..
+my_dir = util
+INSTALL = @INSTALL@
[EMAIL PROTECTED]@
+
+SRCS = $(srcdir)/migrate.c
+
+LIBS= $(LIBEXT2FS) $(LIBCOM_ERR)
+DEPLIBS= $(LIBEXT2FS) $(LIBCOM_ERR)
+
+.c.o:
+ @echo CC $
+ @$(CC) -c $(ALL_CFLAGS) $ -o $@
+ @#cc -g -I../lib/ -Wunreachable-code -Wunused -Wunused-function
+ @#-Wunused-label -Wunused-parameter -Wunused-value -Wunused-variable
-c migrate.c
+
+PROGS= ext4migrate
+
+all:: $(PROGS)
+
+ext4migrate: migrate.o extents.o $(DEPLIBS)
+ @echo LD $@
+ @$(CC) $(ALL_LDFLAGS) -o ext4migrate migrate.o extents.o $(LIBS)
+
+installdirs:
+ @echo MKINSTALLDIRS $(root_sbindir)
+ @$(MKINSTALLDIRS) $(DESTDIR)$(root_sbindir)
+
+install: $(PROGS) installdirs
+ @for i in $(PROGS); do \
+ echo INSTALL $(root_sbindir)/$$i; \
+ $(INSTALL_PROGRAM) $$i $(DESTDIR)$(root_sbindir)/$$i; \
+ done
+
+install-strip: install
+ @for i in $(PROGS); do \
+ echo STRIP $(root_sbindir)/$$i; \
+ $(STRIP) $(DESTDIR)$(root_sbindir)/$$i; \
+ done
+
+uninstall:
+ for i in $(PROGS); do \
+ $(RM) -f $(DESTDIR)$(root_sbindir)/$$i; \
+ done
+clean:
+ $(RM) -f $(PROGS) \#* *.s *.o *.a *~ core
+
+mostlyclean: clean
+
+distclean: clean
+ $(RM) -f .depend Makefile $(srcdir)/TAGS $(srcdir)/Makefile.in.old
+
+# +++ Dependency line eater +++
+#
+# Makefile dependencies follow. This must be the last section in
+# the
[PATCH 2/2] Add extent related functions
From: Aneesh Kumar K.V [EMAIL PROTECTED]
The code is derived out of the latest ext4 kernel source. I
have tried to keep the code as close as possible to the kernel
sources. This makes sure that any fixes for the tree building
code in kernel should be easily applied to ext4migrate. The
ext3_ext naming convention instead of ext4_ext found in kernel is to
make sure we are in sync with rest of e2fsprogs source
Signed-off-by: Aneesh Kumar K.V [EMAIL PROTECTED]
---
ext4migrate/extents.c | 737 +
ext4migrate/extents.h | 10 +
2 files changed, 747 insertions(+), 0 deletions(-)
create mode 100644 ext4migrate/extents.c
create mode 100644 ext4migrate/extents.h
diff --git a/ext4migrate/extents.c b/ext4migrate/extents.c
new file mode 100644
index 000..a21b883
--- /dev/null
+++ b/ext4migrate/extents.c
@@ -0,0 +1,737 @@
+#include stdio.h
+#include stdlib.h
+
+#include migrate.h
+#include extents.h
+
+struct ext4_ext_path {
+ ext4_fsblk_tp_block;
+ __u16 p_depth;
+ struct ext3_extent *p_ext;
+ struct ext3_extent_idx *p_idx;
+ struct ext3_extent_header *p_hdr;
+};
+
+/*
+ * ext_pblock:
+ * combine low and high parts of physical block number into ext4_fsblk_t
+ */
+ext4_fsblk_t ext_pblock(struct ext3_extent *ex)
+{
+ ext4_fsblk_t block;
+
+ block = ex-ee_start;
+ block |= ((ext4_fsblk_t) ex-ee_start_hi 31) 1;
+ return block;
+}
+
+/*
+ * idx_pblock:
+ * combine low and high parts of a leaf physical block number into ext4_fsblk_t
+ */
+ext4_fsblk_t idx_pblock(struct ext3_extent_idx *ix)
+{
+ ext4_fsblk_t block;
+
+ block = ix-ei_leaf;
+ block |= ((ext4_fsblk_t) ix-ei_leaf_hi 31) 1;
+ return block;
+}
+
+/*
+ * ext3_ext_store_pblock:
+ * stores a large physical block number into an extent struct,
+ * breaking it into parts
+ */
+void ext3_ext_store_pblock(struct ext3_extent *ex, ext4_fsblk_t pb)
+{
+ ex-ee_start = (unsigned long) (pb 0x);
+ ex-ee_start_hi = (unsigned long) ((pb 31) 1) 0x;
+}
+
+/*
+ * ext3_idx_store_pblock:
+ * stores a large physical block number into an index struct,
+ * breaking it into parts
+ */
+static void ext3_idx_store_pblock(struct ext3_extent_idx *ix, ext4_fsblk_t pb)
+{
+ ix-ei_leaf = (unsigned long) (pb 0x);
+ ix-ei_leaf_hi = (unsigned long) ((pb 31) 1) 0x;
+}
+
+static int ext3_ext_space_block(ext2_filsys filesys)
+{
+ int size;
+
+ size = (filesys-blocksize - sizeof(struct ext3_extent_header))
+ / sizeof(struct ext3_extent);
+ return size;
+}
+
+static int ext3_ext_space_block_idx(ext2_filsys filesys)
+{
+ int size;
+
+ size = (filesys-blocksize - sizeof(struct ext3_extent_header))
+ / sizeof(struct ext3_extent_idx);
+ return size;
+}
+
+int ext3_ext_space_root(void)
+{
+ int size;
+
+ size = EXT2_N_BLOCKS*sizeof(blk_t);
+ size -= sizeof(struct ext3_extent_header);
+ size /= sizeof(struct ext3_extent);
+ return size;
+}
+
+static int ext3_ext_space_root_idx(void)
+{
+ int size;
+
+ size = EXT2_N_BLOCKS*sizeof(blk_t);
+ size -= sizeof(struct ext3_extent_header);
+ size /= sizeof(struct ext3_extent_idx);
+ return size;
+}
+static void ext3_ext_binsearch_idx(struct ext2_inode *file_inode_struct,
+ struct ext4_ext_path *path, blk_t logical_blk)
+{
+ struct ext3_extent_header *eh = path-p_hdr;
+ struct ext3_extent_idx *r, *l, *m;
+ (void) file_inode_struct;
+
+ l = EXT_FIRST_INDEX(eh) + 1;
+ r = EXT_FIRST_INDEX(eh) + eh-eh_entries - 1;
+ while (l = r) {
+ m = l + (r - l) / 2;
+ if (logical_blk m-ei_block)
+ r = m - 1;
+ else
+ l = m + 1;
+ }
+ path-p_idx = l - 1;
+}
+
+static void ext3_ext_binsearch(struct ext2_inode *file_inode_struct,
+ struct ext4_ext_path *path, blk_t logical_blk)
+{
+ struct ext3_extent_header *eh = path-p_hdr;
+ struct ext3_extent *r, *l, *m;
+ (void) file_inode_struct;
+
+ if (eh-eh_entries == 0) {
+ /*
+* this leaf is empty:
+* we get such a leaf in split/add case
+*/
+ return;
+ }
+
+ l = EXT_FIRST_EXTENT(eh) + 1;
+ r = EXT_FIRST_EXTENT(eh) + eh-eh_entries - 1;
+
+ while (l = r) {
+ m = l + (r - l) / 2;
+ if (logical_blk m-ee_block)
+ r = m - 1;
+ else
+ l = m + 1;
+ }
+
+ path-p_ext = l - 1;
+}
+
+/*
+ * ext3_ext_insert_index:
+ * insert new index [EMAIL PROTECTED];@ptr] into the block at @curp;
+ * check where to insert: before @curp or after @curp
+ */
Re: [RFC][PATCH] sys_fallocate() system call
On Sat, Mar 17, 2007 at 04:33:50PM +1100, Stephen Rothwell wrote: On Fri, 16 Mar 2007 20:01:01 +0530 Amit K. Arora [EMAIL PROTECTED] wrote: +asmlinkage long sys_fallocate(int fd, int mode, loff_t offset, loff_t len); --- linux-2.6.20.1.orig/include/asm-powerpc/systbl.h +++ linux-2.6.20.1/include/asm-powerpc/systbl.h @@ -305,3 +305,4 @@ SYSCALL_SPU(faccessat) COMPAT_SYS_SPU(get_robust_list) COMPAT_SYS_SPU(set_robust_list) COMPAT_SYS(move_pages) +SYSCALL(fallocate) It is going to need to be a COMPAT_SYS call in powerpc because 32 bit powerpc will pass the two loff_t's in pairs of registers while 64bit passes them in one register each. Ok. Will make that change, unless it is decided to pass each loff_t argument as two u32s. Thanks! -- Regards, Amit Arora - To unsubscribe from this list: send the line unsubscribe linux-ext4 in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: e2fsprogs coverity patch cid-04.diff
On Fri, Feb 09, 2007 at 06:11:35PM -0800, Brian D. Behlendorf wrote: This check is unnecessary since fs_type is guaranteed to be set earlier in the function. Coverity ID: 4: Deadcode The code was actually supposed to be dead, so in order to restore the old behavior (from before support for /etc/mke2fs.conf was added) I needed to move the setting of fs_type of journal earlier in the function. - Ted - To unsubscribe from this list: send the line unsubscribe linux-ext4 in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: e2fsprogs coverity patch cid-05.diff
On Fri, Feb 09, 2007 at 06:11:41PM -0800, Brian D. Behlendorf wrote: array is initially set to NULL, so it is possible that readdir() will return NULL and leave array set to NULL. Thus we do need to check if array is NULL or check if num != 0. Coverity ID: 5: Forward NULL This wasn't the only bug; it turns out that if /etc/e2fsck.conf or /etc/mke2fs.conf is an empty directory, then your patch would fix this segfault, only for it to segfault later in profile_init(). I'll fix this before I commit. - Ted - To unsubscribe from this list: send the line unsubscribe linux-ext4 in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: cleaned up ext4 patch series
On Mon, 2007-03-19 at 10:48 -0500, Dave Kleikamp wrote: Ted, I have rebased the ext4 patchset on 2.6.21-rc4 and cleaned up some bad whitespace and sparse warnings. The patches are here: http://www.kernel.org/pub/linux/kernel/people/shaggy/ext4/ext4-2007-03-19.tar.bz2 Untarred here: http://www.kernel.org/pub/linux/kernel/people/shaggy/ext4/ext4-2007-03-19/ I have commented the series file with the changes I've made. Some of the patches are missing signed-of-by: # Rebased the patches to 2.6.21-rc4 # New patch to fix whitespace before applying new patches whitespace.patch # Replaced truncated beginning comments extent-overlap-bugfix persistent_allocation_1_ioctl_and_unitialized_extents We could mention here that this patch is going to be replaced by a new patch to use the fallocate() operations. # Fixed an endian error persistent_allocation_2_support_for_writing_to_unitialized_extent I think Amit has an updated version of this patch in his place. # Note: still lots of outstanding comments from linux-ext4 list, 12/2006 # Missing signed-off-by: booked-page-flag.patch # Missing signed-off-by: ext4-block-reservation.patch # fixed a bunch of endianness errors reported by sparse # Needs a signed-off-by from Alex, then can add shaggy's ext4-delayed-allocation.patch ext4-delalloc-extents-48bit.patch # updated to latest version nanosecond_timestamps.patch This nanosecond patch could be move to upstream earlier than the delayed allocation patch (Alex is working to rewrite it at VFS level), so shall we move it before the booked-page-flag.patch? I wonder if we should create two branches: one branch for patches that are well discussed and tested, which Andrew could trust and pull to mm tree; and create another branch to store patches that are still under discussion and likely to be rewriten based on the review feedback. Thanks, Mingming - To unsubscribe from this list: send the line unsubscribe linux-ext4 in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: cleaned up ext4 patch series
On Mar 19, 2007 09:15 -0800, Mingming Cao wrote: I wonder if we should create two branches: one branch for patches that are well discussed and tested, which Andrew could trust and pull to mm tree; and create another branch to store patches that are still under discussion and likely to be rewriten based on the review feedback. Yes, that makes a lot of sense. It would avoid issues like Andrew finding problems with the nanosecond patches because they hadn't been widely tested yet. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc. - To unsubscribe from this list: send the line unsubscribe linux-ext4 in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] 5/19 e2fsprogs f_extents_ee_len test
Regression test for a bad ee_len field in an extent file. Binary patch file. Signed-off-by: Andreas Dilger [EMAIL PROTECTED] Signed-off-by: Girish Shilamkar [EMAIL PROTECTED] Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc. e2fsprogs-tests-f_extents_ee_len.patch Description: Binary data
[PATCH] 3/19 e2fsprogs f_extents_bad_blk test
Regression test for a bad block in an extent file. Binary patch file. Signed-off-by: Andreas Dilger [EMAIL PROTECTED] Signed-off-by: Girish Shilamkar [EMAIL PROTECTED] Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc. e2fsprogs-tests-f_extents_bad_blk.patch Description: Binary data
[PATCH] 4/19 e2fsprogs f_extents_ee_block test
Regression test for a bad ee_block field in an extent file. Binary patch file. Signed-off-by: Andreas Dilger [EMAIL PROTECTED] Signed-off-by: Girish Shilamkar [EMAIL PROTECTED] Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc. e2fsprogs-tests-f_extents_ee_block.patch Description: Binary data
[PATCH] 6/19 e2fsprogs f_extents_ee_start test
Regression test for a bad ee_start field in an extent file. Binary patch file. Signed-off-by: Andreas Dilger [EMAIL PROTECTED] Signed-off-by: Girish Shilamkar [EMAIL PROTECTED] Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc. e2fsprogs-tests-f_extents_ee_start.patch Description: Binary data
[PATCH] 7/19 e2fsprogs f_extents_eh_depth test
Regression test for a bad eh_depth field in an extent file. Binary patch file. This test still fails with the current extents patch. It takes 2 runs of e2fsck to correct the problem. Signed-off-by: Andreas Dilger [EMAIL PROTECTED] Signed-off-by: Girish Shilamkar [EMAIL PROTECTED] Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc. e2fsprogs-tests-f_extents_eh_depth.patch Description: Binary data
[PATCH] 8/19 e2fsprogs f_extents_eh_entries test
Regression test for a bad eh_entries field in an extent file. Binary patch file. Signed-off-by: Andreas Dilger [EMAIL PROTECTED] Signed-off-by: Girish Shilamkar [EMAIL PROTECTED] Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc. e2fsprogs-tests-f_extents_eh_entries.patch Description: Binary data
[PATCH] 12/19 e2fsprogs f_extents_ei_leaf test
Regression test for a bad ei_leaf field in an extent file. Binary patch file. Signed-off-by: Andreas Dilger [EMAIL PROTECTED] Signed-off-by: Girish Shilamkar [EMAIL PROTECTED] Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc. e2fsprogs-tests-f_extents_ei_leaf.patch Description: Binary data
[PATCH] 14/19 e2fsprogs f_extents_inrlevel-incons test
Regression test for an inter-level index depth inconsistency in an extent file. Binary patch file. Signed-off-by: Andreas Dilger [EMAIL PROTECTED] Signed-off-by: Girish Shilamkar [EMAIL PROTECTED] Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc. e2fsprogs-tests-f_extents_inrlevel-incons.patch Description: Binary data
[PATCH] 15/19 e2fsprogs f_extents_orphan_blks test
Regression test for orphaned blocks in an extent filesystem. Binary patch file. Signed-off-by: Andreas Dilger [EMAIL PROTECTED] Signed-off-by: Girish Shilamkar [EMAIL PROTECTED] Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc. e2fsprogs-tests-f_extents_orphan_blks.patch Description: Binary data
Re: [PATCH] 18/19 e2fsprogs f_extents_shrd_blk test
Regression test for unsorted extents within a file. Binary patch file. Signed-off-by: Andreas Dilger [EMAIL PROTECTED] Signed-off-by: Girish Shilamkar [EMAIL PROTECTED] Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc. e2fsprogs-tests-f_extents_unsorted.patch Description: Binary data
[PATCH] 18/19 e2fsprogs f_extents_shrd_blk test
Regression test for two extent files sharing the same blocks. Binary patch file. This test is known not to work currently. The problem is that there is not enough space in the index to split the extent and clone all of the blocks. It is hoped that Ted's change of e2fsprogs to be extent based instead of block iterator based will allow this problem to be fixed. Signed-off-by: Andreas Dilger [EMAIL PROTECTED] Signed-off-by: Girish Shilamkar [EMAIL PROTECTED] Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc. e2fsprogs-tests-f_extents_shrd_blk.patch Description: Binary data
