Erasure codes
Is HAMMER ever going to have a network based RAID5? Basically implement Erasure Codes?
HAMMER question
Hello, I was taking a peek at http://en.wikipedia.org/wiki/OneFS_distributed_file_system and curious if HAMMER will ever have features like this. TIA
Re: is hammer for us
I was under the impression HAMMER was a parallel filesystem. sorry On Wed, Aug 12, 2009 at 1:04 AM, Matthew Dillondil...@apollo.backplane.com wrote: : :The I/O bottleneck is coming from the disk subsystem and network. I :was wondering if HAMMER can do parallel filesystem implementation :similar to GPFS or Lustre. : :Also, the reads/writes are random access there is very little :sequential streaming, but the files are large.Each file is around 30GB :each It can do master-multi_slave replication if that is what you mean. I don't know how that might compare to GPFS or Lustre. You are going to have more choices in linux-land then in BSD-land, particularly if you have a large array of drives. If there is an I/O bottleneck from the disks due to random access seeks the only solution is more spindles. If you can stage the data in any way a large SSD (solid state drive) might help. e.g. one or more 256G SSDs for data staging eliminates the seek bottlneck and probably also eliminates the need for large amounts of ram in the machines. -Matt
is hammer for us
I am a student doing fluid dynamics research. We generate a lot of data (close to 2TB a day). We are having scalability problems with NFS. We have 2 Linux servers with 64GB of RAM, and they are serving the files. We are constantly running into I/O bottle neck problems. Would hammer fix the scalability problems? TIA
Re: is hammer for us
The I/O bottleneck is coming from the disk subsystem and network. I was wondering if HAMMER can do parallel filesystem implementation similar to GPFS or Lustre. Also, the reads/writes are random access there is very little sequential streaming, but the files are large.Each file is around 30GB each On Tue, Aug 11, 2009 at 11:42 PM, Matthew Dillondil...@apollo.backplane.com wrote: :I am a student doing fluid dynamics research. We generate a lot of :data (close to 2TB a day). We are having scalability problems with :NFS. We have 2 Linux servers with 64GB of RAM, and they are serving :the files. : :We are constantly running into I/O bottle neck problems. Would hammer :fix the scalability problems? : :TIA If you are hitting an I/O bottleneck you need to determine where the bottleneck is. Is it in the actual accesses to the disk subsystem? Are the disks seeking randomly or accessing data linearly? Is the transfer rate acceptable? Is it the network? Is it the NFS implementation? Is it the underlying filesystem on the server? Are there parallelism issues? You need to find the answer to those questions before you can determine a solution. Serving large files typically does not create a filesystem bottleneck. i.e. any filesystem, even something like ZFS, should still be able to serve large linear files at the platter rate. Having a lot of ram only helps if there is some locality of reference in the data set. i.e. if the data set is much larger then available memory but there is no locality of reference and the disk drives are hitting their seek limits, no amount of ram will solve the problem. (DragonFly's 64 bit support isn't reliable yet, so DragonFly can't access that amount of ram right now anyhow). -Matt Matthew Dillon dil...@backplane.com
Re: 1 week until Summer of Code application time
Cool projects. I am rooting for Add redundant data storage to the HAMMER file system. Go Simon :-) On Tue, Mar 3, 2009 at 7:39 AM, Bill Hacker w...@conducive.org wrote: Sdävtaker wrote: It will be amazing if someone can get FreeBSD-UFS mountable (at least for read). I didn't realize that it wasn't Should I cease doing it? OpenBSD is problematic among slices on on same-disk, but even that is apparently resolvable via disklabel editing. Haven't felt the need... yet.. Bill On Tue, Mar 3, 2009 at 01:27, Justin C. Sherrill jus...@shiningsilence.com wrote: Here's a heads-up: Application time for organizations (not students) to get into the 2009 Summer of Code program is 1 week away. If you are any of these things: - potential student - potential mentor - person with an idea for a project Please mark it down at: http://www.dragonflybsd.org/gsoc2009/ It's OK to suggest an idea even if you don't have the time to mentor it. I do need more mentors names - remember, it gets you $500 in addition to helping the DragonFly project a great deal.
Re: HAMMER and RAID 5
Thanks everyone for the replies. Simon good luck with your project, we are all wishing you well! On Tue, Mar 3, 2009 at 3:19 AM, Simon 'corecode' Schubert corec...@fs.ei.tum.de wrote: Mag Gam wrote: I was wondering if HAMMER will ever have network based RAID 5. After researching several file systems it seems HAMMER is probably the closest to achieve this problem and will make HAMMER a pioneer. Any thoughts or ideas? There is a SoC project dealing with local redundancy. I don't think that it will be an overly big deal to run this over network, but it needs more infrastructure then. cheers simon -- 3 the future +++ RENT this banner advert +++ ASCII Ribbon /\ rock the past +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \
HAMMER and RAID 5
I was wondering if HAMMER will ever have network based RAID 5. After researching several file systems it seems HAMMER is probably the closest to achieve this problem and will make HAMMER a pioneer. Any thoughts or ideas? TIA
Hammer question
I am very intrigued with the HAMMER filesystem. I am a heavy Linux user and at work we use Linux exclusively. I was curious how hammer manages dynamic inodes. On ext3 we pre create inodes which is a fixed amount. How is hammer doing this? Sorry if this is a newbie question. I asked the same question on ext3 list and no response there. TIA
Re: Hammer question
Thankyou. I will start dust of my CS books to start looking into Btrees. On Tue, Feb 24, 2009 at 9:58 PM, Matthew Dillon dil...@apollo.backplane.com wrote: :I am very intrigued with the HAMMER filesystem. I am a heavy Linux :user and at work we use Linux exclusively. I was curious how hammer :manages dynamic inodes. On ext3 we pre create inodes which is a fixed :amount. How is hammer doing this? : :Sorry if this is a newbie question. I asked the same question on ext3 :list and no response there. : :TIA Inodes in HAMMER are entries in the B-Tree. They are created and destroyed dynamically. Inode numbers are 64 bit quantities (well, actually 2^63 bits... the positive 64 bit integer space only). Inode numbers in HAMMER cannot be reused for the life of the filesystem. This allows HAMMER to track mirroring (and ultimately cluster) operations regardless of how long mirroring targets are offline. -Matt Matthew Dillon dil...@backplane.com
