Hi, Before doing that, probably one way to think about it could be, what would be the probablitity of two 100M blocks generate the same hash and be treated as identical. Thanks, Xin
Sent: Monday, January 02, 2017 at 4:32 AM From: "Peter Becker" <[email protected]> To: "Xin Zhou" <[email protected]> Cc: linux-btrfs <[email protected]> Subject: Re: [markfasheh/duperemove] Why blocksize is limit to 1MB? > 1M is already a little bit too big in size. Not in my usecase :) Is it right the this isn't an limit in btrfs? So i can patch this and try 100M. The reason is, that i must dedupe the whole 8 TB in less then a day but with 128K and 1M blocksize it will take a week. I don't know why adding extends take so long. I/O during adding extends is less then 4MB/s, and CPU (dual core) and memory (8 GB) usage are less then 20%, on bare metal. 2017-01-01 5:38 GMT+01:00 Xin Zhou <[email protected]>: > Hi, > > In general, the larger the block / chunk size is, the less dedup can be > achieved. > 1M is already a little bit too big in size. > > Thanks, > Xin > > > > > Sent: Friday, December 30, 2016 at 12:28 PM > From: "Peter Becker" <[email protected]> > To: linux-btrfs <[email protected]> > Subject: [markfasheh/duperemove] Why blocksize is limit to 1MB? > Hello, i have a 8 TB volume with multiple files with hundreds of GB each. > I try to dedupe this because the first hundred GB of many files are identical. > With 128KB blocksize with nofiemap and lookup-extends=no option, will > take more then a week (only dedupe, previously hashed). So i tryed -b > 100M but this returned me an error: "Blocksize is bounded ...". > > The reason is that the blocksize is limit to > > #define MAX_BLOCKSIZE (1024U*1024) > > But i can't found any description why. > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to [email protected] > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
