Re: [PATCH 00/25] Btrfs-convert rework to support native separate
On Thu, Nov 26, 2015 at 08:38:23AM +0800, Qu Wenruo wrote: > > As far as the conversion support stays, it's not a problem of course. I > > don't have a complete picture of all the actual merging conflicts, but > > the idea is to provide the callback abstraction v2 to allow ext2 and > > reiser plus allow all the changes of this pathcset. > > > Glad to hear that. > > BTW, which reiserfs progs headers are you using? Sorry I forgot to mention it, it's the latest git version, https://git.kernel.org/cgit/linux/kernel/git/jeffm/reiserfsprogs.git/ Jeff hasn't released v3.6.25 yet. We have the git version in SUSE distros so it works for me here. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 00/25] Btrfs-convert rework to support native separate
On 11/26/2015 05:30 PM, David Sterba wrote: On Thu, Nov 26, 2015 at 08:38:23AM +0800, Qu Wenruo wrote: As far as the conversion support stays, it's not a problem of course. I don't have a complete picture of all the actual merging conflicts, but the idea is to provide the callback abstraction v2 to allow ext2 and reiser plus allow all the changes of this pathcset. Glad to hear that. BTW, which reiserfs progs headers are you using? Sorry I forgot to mention it, it's the latest git version, https://git.kernel.org/cgit/linux/kernel/git/jeffm/reiserfsprogs.git/ Jeff hasn't released v3.6.25 yet. We have the git version in SUSE distros so it works for me here. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Thanks, now it should be OK to continue the rebase. But I'm a little concerned about the unstable headers, unlike ext2 its headers is almost stable but reiserfs seems not. What about rebasing my patch to your abstract patch (btrfs-progs: convert: add context and operations struct to allow different file systems) first and add back your reiserfs patch? Your abstract patch is quite nice, although need some modification to work with new convert. I hope to add stable things first and don't want another reiserfs change breaks the compile. Thanks, Qu -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 00/25] Btrfs-convert rework to support native separate
On Thu, Nov 26, 2015 at 06:12:57PM +0800, Qu Wenruo wrote: > But I'm a little concerned about the unstable headers, unlike ext2 its > headers is almost stable but reiserfs seems not. Well, reiserfs is not developped nowadays and I think Jeff implemented the bits required for btrfs-convert. The configure script will detect if the reiser library provided needed functions, compiling reiser in will be optional anyway. > What about rebasing my patch to your abstract patch (btrfs-progs: > convert: add context and operations struct to allow different file > systems) first and add back your reiserfs patch? Oh right, the patch is independent, I'll add it to devel. > Your abstract patch is quite nice, although need some modification to > work with new convert. Yes, that's expected. > I hope to add stable things first and don't want another reiserfs change > breaks the compile. Ok. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 00/25] Btrfs-convert rework to support native separate
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 11/26/15 5:12 AM, Qu Wenruo wrote: > > > On 11/26/2015 05:30 PM, David Sterba wrote: >> On Thu, Nov 26, 2015 at 08:38:23AM +0800, Qu Wenruo wrote: As far as the conversion support stays, it's not a problem of course. I don't have a complete picture of all the actual merging conflicts, but the idea is to provide the callback abstraction v2 to allow ext2 and reiser plus allow all the changes of this pathcset. >>> Glad to hear that. >>> >>> BTW, which reiserfs progs headers are you using? >> >> Sorry I forgot to mention it, it's the latest git version, >> https://git.kernel.org/cgit/linux/kernel/git/jeffm/reiserfsprogs.git/ >> >> >> Jeff hasn't released v3.6.25 yet. We have the git version in SUSE >> distros so it works for me here. -- To unsubscribe from this >> list: send the line "unsubscribe linux-btrfs" in the body of a >> message to majord...@vger.kernel.org More majordomo info at >> http://vger.kernel.org/majordomo-info.html >> > Thanks, now it should be OK to continue the rebase. > > But I'm a little concerned about the unstable headers, unlike ext2 > its headers is almost stable but reiserfs seems not. This is entirely due to the fact that splitting out library functionality for reiserfsprogs was only done to support btrfs-convert. Unless there's some pressing need to revise the API, the headers are pretty much static at this point. I should just go ahead and release the current snapshot as 3.6.25. Today's a US holiday and I won't be able to get to it, but I'll do that in the next few days. - -Jeff > What about rebasing my patch to your abstract patch (btrfs-progs: > convert: add context and operations struct to allow different file > systems) first and add back your reiserfs patch? > > Your abstract patch is quite nice, although need some modification > to work with new convert. I hope to add stable things first and > don't want another reiserfs change breaks the compile. > > Thanks, Qu > - -- Jeff Mahoney SUSE Labs -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2 iQIcBAEBCAAGBQJWVyECAAoJEB57S2MheeWyHFQP/jNSZthnme+Gr6HV9DcGNuWb F3m4HOauGny+5mWzWjzATcS6YjAB0Hr0ObXi6jgoQxueBTInZKfgRYIqF6q1hdxS NymluoAi9lkTkgkiCTZWmUexUaGE2pDzWR14dzqkBorqfCkvyvBVrkXV32FQUJ9H ln85QND805HUiHol3+rqSSFhxN8A8C+3UMkxOuUCrlkBx8KkzdKVrFJNH4+2L3Ml tt1KYkWa4hCOYqFivzxDJ69HWNBkUIPXsig+38Lw/JkuLbt82DfMAVu5QbttAgus 1/ZOhPwv7grRfV/CpCpNHPV/AvLLLR0wu5DNCej4HsC81WH4KUE5Isavk2WxQAF3 gpFdnxh2Ok3n+g/obFpDKh43XjVIbtRel4bsB13WjfM9mUNFX8lm/vnK8bjmuoGo FRf0eQwY3YZAzFrTglFBQxK4eUnopDr1CTFTmnLtBoYKSojDdwe8KQTDWuimLyXb /BY7qeiW3Cahm+V2VrL76cnbxg5/xH0u6CKrfsg9p4NpP9+5bi72vLgy16IJI+a2 jf6pLvgOi9MwMGW42tIHGna+vFwPAaoL4Iqm4qbWqHZ/R91bNAfWgOdBEubqHuYi qML5RAttojrw7ZtwSLkBxnOyj3ias3c2jDTQ1yqaSVMJ6phn9XwzB5v4I87qEdsS a8QhP69u++S+PcohdSYQ =3s6k -END PGP SIGNATURE- -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 00/25] Btrfs-convert rework to support native separate
David Sterba wrote on 2015/11/25 13:42 +0100: On Tue, Nov 24, 2015 at 04:50:00PM +0800, Qu Wenruo wrote: It seems the conflict is quite huge, your reiserfs support is based on the old behavior, just like what old ext2 one do: custom extent allocation. I'm afraid the rebase will take a lot of time since I'm completely a newbie about reiserfs... :( Yeah, the ext2 callbacks are abstracted and replaced by reiserfs implementations, and the abstratction is quite direct. This might be a problem with merging your patchset. The abstraction is better than I expected, and should be quite handle to use. Although a lot of my codes will be changed to use it. I may need to change a lot of ext2 direct call to generic one, and may even change the generic function calls.(no alloc/free, only free space lookup) And some (maybe a lot) of reiserfs codes may be removed during the rework. As far as the conversion support stays, it's not a problem of course. I don't have a complete picture of all the actual merging conflicts, but the idea is to provide the callback abstraction v2 to allow ext2 and reiser plus allow all the changes of this pathcset. Glad to hear that. BTW, which reiserfs progs headers are you using? It seems that the headers you are using is quite different from what my distribution is providing, and this makes compile impossible. For example, in my /usr/include/reiserfs, there is no io.h, no reiserfs.h. No structure named reseifs_key, but only key. Not sure if it is my progsreiserfs is too old or whatever other reason. What progsreiserfs are you using? Thanks, Qu -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 00/25] Btrfs-convert rework to support native separate
On Tue, Nov 24, 2015 at 04:50:00PM +0800, Qu Wenruo wrote: > It seems the conflict is quite huge, your reiserfs support is based on > the old behavior, just like what old ext2 one do: custom extent allocation. > I'm afraid the rebase will take a lot of time since I'm completely a > newbie about reiserfs... :( Yeah, the ext2 callbacks are abstracted and replaced by reiserfs implementations, and the abstratction is quite direct. This might be a problem with merging your patchset. > I may need to change a lot of ext2 direct call to generic one, and may > even change the generic function calls.(no alloc/free, only free space > lookup) > > And some (maybe a lot) of reiserfs codes may be removed during the rework. As far as the conversion support stays, it's not a problem of course. I don't have a complete picture of all the actual merging conflicts, but the idea is to provide the callback abstraction v2 to allow ext2 and reiser plus allow all the changes of this pathcset. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 00/25] Btrfs-convert rework to support native separate
David Sterba wrote on 2015/11/23 18:33 +0100: On Fri, Nov 20, 2015 at 11:24:04AM +0800, Qu Wenruo wrote: Here comes the 1st version of btrfs-convert rework. Any test is welcomed, and it can already pass the convert test from btrfs-progs. (Since the test doesn't test rollback function) I went through the patches, looks mostly ok akin to the proposed changes. I'll take the independent patches rightaway. Unfortunatelly there are code changes that clash significantly with the pending reiserfs addition to convert, I'm afraid you'll have to rework your patchset on top of that. The code is now pushed to branch 'dev/convert-reiser'. Hi David, It seems the conflict is quite huge, your reiserfs support is based on the old behavior, just like what old ext2 one do: custom extent allocation. I'm afraid the rebase will take a lot of time since I'm completely a newbie about reiserfs... :( I may need to change a lot of ext2 direct call to generic one, and may even change the generic function calls.(no alloc/free, only free space lookup) And some (maybe a lot) of reiserfs codes may be removed during the rework. Will it be OK for you? Thanks, Qu -- This message has been scanned for viruses and dangerous content by Fujitsu, and is believed to be clean. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 00/25] Btrfs-convert rework to support native separate
David Sterba wrote on 2015/11/23 18:33 +0100: On Fri, Nov 20, 2015 at 11:24:04AM +0800, Qu Wenruo wrote: Here comes the 1st version of btrfs-convert rework. Any test is welcomed, and it can already pass the convert test from btrfs-progs. (Since the test doesn't test rollback function) I went through the patches, looks mostly ok akin to the proposed changes. I'll take the independent patches rightaway. Unfortunatelly there are code changes that clash significantly with the pending reiserfs addition to convert, I'm afraid you'll have to rework your patchset on top of that. The code is now pushed to branch 'dev/convert-reiser'. Thanks for the check, David. I'll rebase it soon. Thanks, Qu -- This message has been scanned for viruses and dangerous content by FCNIC, and is believed to be clean. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 00/25] Btrfs-convert rework to support native separate
On Fri, Nov 20, 2015 at 11:24:04AM +0800, Qu Wenruo wrote: > Here comes the 1st version of btrfs-convert rework. > Any test is welcomed, and it can already pass the convert test from > btrfs-progs. (Since the test doesn't test rollback function) I went through the patches, looks mostly ok akin to the proposed changes. I'll take the independent patches rightaway. Unfortunatelly there are code changes that clash significantly with the pending reiserfs addition to convert, I'm afraid you'll have to rework your patchset on top of that. The code is now pushed to branch 'dev/convert-reiser'. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 00/25] Btrfs-convert rework to support native separate
Here comes the 1st version of btrfs-convert rework. Any test is welcomed, and it can already pass the convert test from btrfs-progs. (Since the test doesn't test rollback function) I also did some tests like create ext4 and fill it with fsstress to test convert, at least no bug spotted yet. The new btrfs-convert has the following bugfix/feature: 1. True separate meta/data chunk The converted filesystem will has correct chunk layout, and all ext* data will be covered by data chunk and metadata will be covered by metadata chunk. Allowing converted filesystem to pass latest btrfs check. 2. Structurized make_btrfs_v2() Unlike old make_btrfs() which is a mess of hand written codes, new make_btrfs_v2() is all structurized and should be easy for later expansion. But it still has something to do: 1) Ability to handle 10+ TB level filesystem. Current implement rely on the initial system chunk to contain all chunks until we inserted all data chunks. However the initial system chunk are only 32M, in simple calculation, it can only hold at most 400K chunks. If every chunk is in the smallest size (16M), it can only handle less than 6T size in its worst case. Better calculate the initial system chunk size according to the filesystem size 2) Ability to create DUP metadata chunk without using balance Now the data and metadata chunk are all in SINGLE profile. User needs to convert metadata to DUP if they want DUP metadata profile. This is a little hard, as we must do it at make_btrfs() time, where we don't have full btrfs facilities to modify the metadata tree. This may need another rework on functions like btrfs_search_slot() to support temporary btrfs image without btrfs_root structure. The new btrfs-convert will work like the following: 1) Scan ext2 for all used space 2) Calculate the following tree maps: convert_data_chunks: Ranges must be covered by data chunks Maps are batched to avoid small chunk and avoid reserved ranges like the first 1MB of the device and superblock space. free_space: Ranges that later allocation can allocate from With reserved ranges wiped. 3) Reserved space for superblock and system/metadata chunks Use free_space above to reserve space 4) Make a temporary btrfs, using above super/system/metadata position This is done by the new make_btrfs_v2() function. 5) Open temporary btrfs 6) Insert all data chunks to cover all ext* data 7) Create ext2_save subvolume and image Unlike old implement, which did this later, we do this first, as this method can reduce the extent fragments, reducing extent tree size. And this also calculate all the csum if needed, making later inode copy free from calculating csum. 8) Handle reserved ranges Copy ext* data in reserved ranges into other places. Now the hole ext2_save subvolume and image is created. 9) Copy inodes Inode copy will use the ext2_save image as new logical <-> disk mapping, to handle reserved ranges. For rollback function, since the only difference part lies in how we handle reserved ranges, the modification is quite small, and it will continue supporting old behavior btrfs-convert for compatibility. Changelog: v2: Add rollback support Make btrfs-convert more safe on chunk type Qu Wenruo (25): btrfs-progs: extent-cache: Add comments for search/lookup functions btrfs-progs: extent-tree: Add add_merge_cache_extent function btrfs-progs: Add new init/free function and member for mkfs_config btrfs-progs: convert: Read and build up used space tree btrfs-progs: utils: Introduce new function to remove reserved ranges btrfs-progs: utils: Introduce function to calculate the available space btrfs-progs: Reserve space for system/meta chunks and superblock btrfs-progs: Introduce function to setup temporary superblock btrfs-progs: Introduce function to setup temporary tree root btrfs-progs: Introduce function to setup temporary chunk root btrfs-progs: Introduce function to initialize device tree btrfs-progs: Introduce function to initialize fs tree btrfs-progs: Introduce function to initialize csum tree btrfs-progs: Introduce function to setup temporary extent tree btrfs-progs: Introduce function to create convert data chunks btrfs-progs: extent-tree: Introduce function to find the first overlap extent. btrfs-progs: extent-tree: Enhance btrfs_record_file_extent btrfs-progs: convert: Introduce new function to create ext2 image btrfs-progs: convert: Introduce function to migrate reserved ranges btrfs-progs: Enhance record_file_blocks to handle reserved ranges btrfs-progs: convert: Introduce init_btrfs_v2 function. btrfs-progs: Introduce do_convert_v2 function btrfs-progs: Convert: Add support for rollback new convert behavior btrfs-progs: convert: Strictly avoid meta or system chunk allocation btrfs-progs: Cleanup old btrfs-convert btrfs-convert.c | 1617
