Re: [RFC][take 4] e2fsprogs: Add ext4migrate
On Mon, May 07, 2007 at 08:56:11AM +0530, Aneesh Kumar K.V wrote: Andreas Dilger wrote: If this code could also (or optionally just) increase the size of inodes it would be very useful. AFAIK right now it will only change the inodes from block-mapped to extent-mapped? Actually, changing the inode size is actually going to be more useful to a larger number of people, since they can use it today to support in-inode EA's. In addition, changing the inode size must be done off-line, whereas it's not so clear that off-line conversion to extents is the best way to go (more on that below). One generic concern I have about simply migrating existing files to use extents is that unless we actually also combine the tool with defragmentation support, the file ends up being fairly badly fragmented simply because of the holes left in the file from the indirect blocks. Therefore the tree ends up being far larger than it needs to be, and future attempts allocate blocks in that block group will fill in the holes, further degrading the performance of the filesystem and whatever file is being written at the time. The other reason why increasing the inode size would actually be more valuable is that right now, if we have the disk space, the user can do on-line migration of a file simply by copying it and then moving it over the original file --- and if we have delayed allocation support and sufficient memory, we can avoid the fragmentation problems currently with the off-line ext4migration approach. So we may need to think a bit about what's the best way to handle ext4migrate. It may be that correct approach is to combine it with a defragmentation tool, either on-line or off-line. That is correct. My next step is to enhance the tool to support migration to large inodes. I would actually suggest that the place to add that functionality would be via tune2fs -I inode_size. Since mke2fs -I inode_size is used to set the inode size, it makes sense that tune2fs -I would change the inode size. Obviously this would have to be done when the filesystem is not mounted, and tune2fs should abort gracefully if determines that the filesystem is mounted. - Ted - To unsubscribe from this list: send the line unsubscribe linux-ext4 in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC][take 4] e2fsprogs: Add ext4migrate
Hi, Theodore Tso wrote: On Mon, May 07, 2007 at 08:56:11AM +0530, Aneesh Kumar K.V wrote: Andreas Dilger wrote: If this code could also (or optionally just) increase the size of inodes it would be very useful. AFAIK right now it will only change the inodes from block-mapped to extent-mapped? Actually, changing the inode size is actually going to be more useful to a larger number of people, since they can use it today to support in-inode EA's. In addition, changing the inode size must be done off-line, whereas it's not so clear that off-line conversion to extents is the best way to go (more on that below). One of the option i was thinking was to use this tool to migrate to extent map and then towards the end use the online defrag ioctl to defrag the resulting ext4 inode. That way we don't need to add kernel code that convert ext3 inode to ext4 inode while defragmentation. Last time i looked at the online defrag code, i was not able to figure out a easy way to take indirect map inode as input, then defrag the same and write the result in extent map. -aneesh - To unsubscribe from this list: send the line unsubscribe linux-ext4 in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC][take 4] e2fsprogs: Add ext4migrate
Theodore Tso wrote: That is correct. My next step is to enhance the tool to support migration to large inodes. I would actually suggest that the place to add that functionality would be via tune2fs -I inode_size. Since mke2fs -I inode_size is used to set the inode size, it makes sense that tune2fs -I would change the inode size. Obviously this would have to be done when the filesystem is not mounted, and tune2fs should abort gracefully if determines that the filesystem is mounted. That is nice !!. I will add the large inode migration support to tune2fs. -aneesh - To unsubscribe from this list: send the line unsubscribe linux-ext4 in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC][take 4] e2fsprogs: Add ext4migrate
On Mon, May 07, 2007 at 07:16:31PM +0530, Aneesh Kumar K.V wrote: One of the option i was thinking was to use this tool to migrate to extent map and then towards the end use the online defrag ioctl to defrag the resulting ext4 inode. As we discussed on the phone, my recommendation would be to take your existing code and move it into the kernel so that triggered off of an ioctl, your code could migrate an inode from using indirect blocks to extents while a filesystem is mounted. The main things you will need to watch for is to make sure the inode is locked so that another process doesn't try to extend or truncate it, and to use the JBD layer to provide appropriate journalling support. Given that ext4migrated imported the kernel extent functions, it should be relatively straightforward to simply make them use the kernel extent functions while in kernel space. Once the the inode has been converted on-line then it can be defragmented on-line. That would be much more convenient than having to unmount the filesystem to do the off-line migration, followed by mounting it to do the defragmentation. Regards, - Ted - To unsubscribe from this list: send the line unsubscribe linux-ext4 in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC][take 4] e2fsprogs: Add ext4migrate
On May 07, 2007 19:16 +0530, Aneesh Kumar K.V wrote: Theodore Tso wrote: Actually, changing the inode size is actually going to be more useful to a larger number of people, since they can use it today to support in-inode EA's. In addition, changing the inode size must be done off-line, whereas it's not so clear that off-line conversion to extents is the best way to go (more on that below). That way we don't need to add kernel code that convert ext3 inode to ext4 inode while defragmentation. Last time i looked at the online defrag code, i was not able to figure out a easy way to take indirect map inode as input, then defrag the same and write the result in extent map. I think the online defrag code _should_ be fairly easily made to handle defrag of block-mapped files, so long as it is always creating extent- mapped files in the end. All that should be necessary is to have the kernel read data from the block-mapped file and write it into the newly mapped blocks. That functionality needs to exist for a long time anyways to support upgrading the filesystem so it shouldn't be a huge burden. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc. - To unsubscribe from this list: send the line unsubscribe linux-ext4 in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC][take 4] e2fsprogs: Add ext4migrate
Andreas Dilger wrote: On May 04, 2007 14:43 +0530, Aneesh Kumar K.V wrote: Changes from from my previous patches are as follows: a) support for files with holes b) use the block iterator present in libext2fs c) don't mark the indirect blocks as unused early. If we mark the the blocks unused, they can be resued for extent index. If we later fail to migrate, the indirect blocks content would be overwritten. If this code could also (or optionally just) increase the size of inodes it would be very useful. AFAIK right now it will only change the inodes from block-mapped to extent-mapped? That is correct. My next step is to enhance the tool to support migration to large inodes. -aneesh - To unsubscribe from this list: send the line unsubscribe linux-ext4 in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC][take 4] e2fsprogs: Add ext4migrate
This is work in progress Changes from from my previous patches are as follows: a) COW unix io manager using libtdb b) add ext4replay to undo the file system changes done during migration Changes from from my previous patches are as follows: a) Mark the file system unclean if we fail migrating b) support for migrating more than one file c) Support for migrating all the ext3 inode in the file system d) Fix the free block counts. ( Make sure fsck pass without errors after migration ) e) Mark the blocks used for extent idx as used so that we don't double allocate them f) Don't mark the blocks used for indirect block map as free untill we fully migrate the inode.This make sure we don't use these blocks for extent idx and overwrite them. g) Fix the migration when we end up with more than one extent idx block. Changes from from my previous patches are as follows: a) support for files with holes b) use the block iterator present in libext2fs c) don't mark the indirect blocks as unused early. If we mark the the blocks unused, they can be resued for extent index. If we later fail to migrate, the indirect blocks content would be overwritten. ext4migrate command takes the below syntax ext4migrate --display | --migrate image_name [filename] The --display option helps in displaying the block map details for an ext3 inode and extent map details for an ext4 inode. The --migrate option convert a block mapped ext3 inode to extent mapped ext4 inode. This needs to be run on an unmounted file system (offline migration). The extent insert code is derived out of the latest ext4 kernel source. I have tried to keep the code as close as possible to the kernel sources. This makes sure that any fixes for the tree building code in kernel should be easily applied to ext4migrate. The ext3_ext naming convention instead of ext4_ext found in kernel is to make sure we are in sync with rest of e2fsprogs source. The inode modification is done only at the last stage. This is to make sure that if we fail at any intermediate stage, we exit without touching the disk. The inode update is done as below a) Walk the extent index blocks and write them to the disk. If failed exit b) Writ the update block bitmap. if failed exit. b) Write the inode. if failed Undo the write of block bitmap and exit. The patch applies on top of e2fsprogs found at http://www.kernel.org/pub/linux/kernel/people/tytso/e2fsprogs-interim/e2fsprogs-1.39-tyt1/e2fsprogs-1.39-tyt1.tar.bz2 Right now i am building ext4migrate and ext4replay by linking against libtd. This will not be needed once I rebase the patches against the latest e2fsprogs which has the tdb code available as a part libext2fs -aneesh - To unsubscribe from this list: send the line unsubscribe linux-ext4 in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
