Re: ChunkFS - measuring cross-chunk references

2007-05-19 Thread Karuna sagar K
On 4/25/07, Suparna Bhattacharya [EMAIL PROTECTED] wrote: On Wed, Apr 25, 2007 at 05:50:55AM +0530, Karuna sagar K wrote: One more set of numbers to calculate would be an estimate of cross-references across chunks of block groups -- 1 (=128MB), 2 (=256MB), 4 (=512MB), 8(=1GB) as suggested by

Re: ChunkFS - measuring cross-chunk references

2007-05-06 Thread Valerie Henson
On Mon, Apr 23, 2007 at 02:53:33PM -0600, Andreas Dilger wrote: Also, is it considered a cross-chunk reference if a directory entry is referencing an inode in another group? Should there be a continuation inode in the local group, or is the directory entry itself enough? (Sorry for the

Re: ChunkFS - measuring cross-chunk references

2007-05-06 Thread Valerie Henson
On Mon, Apr 23, 2007 at 02:05:47AM +0530, Karuna sagar K wrote: Hi, The attached code contains program to estimate the cross-chunk references for ChunkFS file system (idea from Valh). Below are the results: Nice work! Thank you very much for doing this! -VAL - To unsubscribe from this

Re: ChunkFS - measuring cross-chunk references

2007-05-06 Thread Valerie Henson
On Mon, Apr 23, 2007 at 08:13:06PM -0400, Theodore Tso wrote: There may also be special things we will need to do to handle scenarios such as BackupPC, where if it looks like a directory contains a huge number of hard links to a particular chunk, we'll need to make sure that directory is

Re: ChunkFS - measuring cross-chunk references

2007-04-25 Thread Suparna Bhattacharya
On Wed, Apr 25, 2007 at 05:50:55AM +0530, Karuna sagar K wrote: On 4/24/07, Theodore Tso [EMAIL PROTECTED] wrote: On Mon, Apr 23, 2007 at 02:53:33PM -0600, Andreas Dilger wrote: . It would also be good to distinguish between directories referencing files in another chunk, and

Re: ChunkFS - measuring cross-chunk references

2007-04-24 Thread Theodore Tso
On Mon, Apr 23, 2007 at 06:02:29PM -0700, Arjan van de Ven wrote: The other thing which we should consider is that chunkfs really requires a 64-bit inode number space, which means either we only allow does it? I'd think it needs a chunk space number and a 32 bit local inode number ;)

Re: ChunkFS - measuring cross-chunk references

2007-04-24 Thread Karuna sagar K
On 4/24/07, Theodore Tso [EMAIL PROTECTED] wrote: On Mon, Apr 23, 2007 at 02:53:33PM -0600, Andreas Dilger wrote: . It would also be good to distinguish between directories referencing files in another chunk, and directories referencing subdirectories in another chunk (which would be

Re: ChunkFS - measuring cross-chunk references

2007-04-23 Thread Karuna sagar K
Hi, The tool estimates the cross-chunk references from an extt2/3 file system. It considers a block group as one chunk and calcuates how many block groups does a file span across. So, the block group size gives the estimate of chunk size. The file systems were aged for about 3-4 months on a

Re: ChunkFS - measuring cross-chunk references

2007-04-23 Thread Andreas Dilger
On Apr 23, 2007 15:04 +0530, Kalpak Shah wrote: On Mon, 2007-04-23 at 12:49 +0530, Karuna sagar K wrote: The tool estimates the cross-chunk references from an extt2/3 file system. It considers a block group as one chunk and calcuates how many block groups does a file span across. So, the

Re: ChunkFS - measuring cross-chunk references

2007-04-23 Thread Theodore Tso
On Mon, Apr 23, 2007 at 02:53:33PM -0600, Andreas Dilger wrote: With a blocksize of 4KB, a block group would be 128 MB. In the original Chunkfs paper, Valh had mentioned 1GB chunks and I believe it will be possible to use 2GB, 4GB or 8GB chunks in the future. As the chunk size increases

Re: ChunkFS - measuring cross-chunk references

2007-04-23 Thread Arjan van de Ven
The other thing which we should consider is that chunkfs really requires a 64-bit inode number space, which means either we only allow does it? I'd think it needs a chunk space number and a 32 bit local inode number ;) (same for blocks) - To unsubscribe from this list: send the line

Re: ChunkFS - measuring cross-chunk references

2007-04-23 Thread Amit Gud
On Mon, 23 Apr 2007, Arjan van de Ven wrote: The other thing which we should consider is that chunkfs really requires a 64-bit inode number space, which means either we only allow does it? I'd think it needs a chunk space number and a 32 bit local inode number ;) (same for blocks) For

Re: ChunkFS - measuring cross-chunk references

2007-04-23 Thread Amit Gud
On Mon, 23 Apr 2007, Amit Gud wrote: On Mon, 23 Apr 2007, Arjan van de Ven wrote: The other thing which we should consider is that chunkfs really requires a 64-bit inode number space, which means either we only allow does it? I'd think it needs a chunk space number and a 32 bit local

Re: ChunkFS - measuring cross-chunk references

2007-04-22 Thread Amit Gud
Karuna sagar K wrote: Hi, The attached code contains program to estimate the cross-chunk references for ChunkFS file system (idea from Valh). Below are the results: Nice to see some numbers! But would be really nice to know: - what the chunk size is - how the files were created or, more