Re: Anyone working on fsck?

2003-03-18 Thread Brad Knowles
At 8:32 PM -0800 2003/03/17, Terry Lambert wrote: Even so, for RAID, this is generally problematic, because there's multiple locations for the block: where it lives, where it's mirrored, where it's parity block lives, etc.. Ideally, these are all different spindles, so the problem can't be

Re: Anyone working on fsck?

2003-03-18 Thread Brad Knowles
At 10:49 PM -0800 2003/03/17, Terry Lambert wrote: Yes, I know. I'm aware. He has a lot of data to transfer in from the disk in order to do the reverse lookup, with that much data on the disk. I'm confused. In situations where you need to do reverse lookups, don't you normally tend to

Re: Anyone working on fsck?

2003-03-18 Thread Brad Knowles
At 7:17 AM +0100 2003/03/18, Poul-Henning Kamp wrote: Optimizing fsck is a valid project, I just wish it would be somebody who would also finish the last 30% who would do it. Just what are you saying? Is Julian Elischer not the right person to be working on this, because he has a history

Re: Anyone working on fsck?

2003-03-18 Thread Giorgos Keramidas
On 2003-03-17 14:05, Bakul Shah [EMAIL PROTECTED] wrote: If you start to implement any sort of journaling (that is what you talked about in your email), you might as well just stop right at the clean bit, and avoid the complexity. No, I didn't suggest journaling, I suggested storing all state

Re: Anyone working on fsck?

2003-03-18 Thread Terry Lambert
Brad Knowles wrote: At 10:49 PM -0800 2003/03/17, Terry Lambert wrote: Yes, I know. I'm aware. He has a lot of data to transfer in from the disk in order to do the reverse lookup, with that much data on the disk. I'm confused. In situations where you need to do reverse

Re: Anyone working on fsck?

2003-03-18 Thread Brad Knowles
At 2:42 PM -0800 2003/03/18, Terry Lambert wrote: Make sense now? No. However, I am now convinced that I don't understand enough of how the filesystem works to even be able to ask the simplest of questions about how this process can be improved. So, I will now shut up. -- Brad Knowles,

Re: Anyone working on fsck?

2003-03-18 Thread Terry Lambert
Brad Knowles wrote: At 2:42 PM -0800 2003/03/18, Terry Lambert wrote: Make sense now? No. However, I am now convinced that I don't understand enough of how the filesystem works to even be able to ask the simplest of questions about how this process can be improved. So,

Re: Anyone working on fsck?

2003-03-18 Thread Dag-Erling Smørgrav
Julian Elischer [EMAIL PROTECTED] writes: [fsck is impossibly slow on multi-TB filesystems] Let's rather work on getting a working log-structured filesystem committed so we don't *need* fsck for filesystems that large. DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail

Re: Anyone working on fsck?

2003-03-18 Thread Miguel Mendez
On Tue, 18 Mar 2003 10:59:14 +0100 [EMAIL PROTECTED] (Dag-Erling Smørgrav) wrote: Howdy, Let's rather work on getting a working log-structured filesystem committed so we don't *need* fsck for filesystems that large. Rather than reinventing the wheel, how about porting NetBSD's LFS to

Anyone working on fsck?

2003-03-17 Thread Julian Elischer
Is there anyone working on fsck? Recent timings with a fast machine with 1TB filesystems show that it takes abuot 6 hours to fsck such a filesystem (on a fast array with a lot of RAM) This is with a version of fsck that already has some locally developed speedups and changes. I have not dared

Re: Anyone working on fsck?

2003-03-17 Thread Alfred Perlstein
* Julian Elischer [EMAIL PROTECTED] [030317 12:22] wrote: Is there anyone working on fsck? Recent timings with a fast machine with 1TB filesystems show that it takes abuot 6 hours to fsck such a filesystem (on a fast array with a lot of RAM) This is with a version of fsck that already has

Re: Anyone working on fsck?

2003-03-17 Thread Julian Elischer
this is a full 100% forground fsck -y On Mon, 17 Mar 2003, Alfred Perlstein wrote: * Julian Elischer [EMAIL PROTECTED] [030317 12:22] wrote: Is there anyone working on fsck? Recent timings with a fast machine with 1TB filesystems show that it takes abuot 6 hours to fsck

Re: Anyone working on fsck?

2003-03-17 Thread Julian Elischer
: * Julian Elischer [EMAIL PROTECTED] [030317 12:22] wrote: Is there anyone working on fsck? Recent timings with a fast machine with 1TB filesystems show that it takes abuot 6 hours to fsck such a filesystem (on a fast array with a lot of RAM) This is with a version of fsck

Re: Anyone working on fsck?

2003-03-17 Thread Miguel Mendez
On Mon, 17 Mar 2003 12:22:33 -0800 (PST) Julian Elischer [EMAIL PROTECTED] wrote: Howdy, It wouldn;t be super fast but at least it COULD be used to check a 30TB array, where the in-memory version would beed a process VM space of 24MB which is clearly impossible on a x86. I'm sure most

Re: Anyone working on fsck?

2003-03-17 Thread Bakul Shah
UFS is the real problem here, not fsck. Its tradeoffs for improving normal access latencies may have been right in the past but not for modern big disks. The seek time RPM have not improved very much in the past 20 years while disk capacity has increased by a factor of about 20,000 (and GB/$

Re: Anyone working on fsck?

2003-03-17 Thread Poul-Henning Kamp
In message [EMAIL PROTECTED], Bakul Shah writes: UFS is the real problem here, not fsck. Its tradeoffs for improving normal access latencies may have been right in the past but not for modern big disks. The seek time RPM have not improved very much in the past 20 years while disk capacity has

Re: Anyone working on fsck?

2003-03-17 Thread Julian Elischer
Thanks for your thoughts. . Some good points.. On Mon, 17 Mar 2003, Bakul Shah wrote: UFS is the real problem here, not fsck. Its tradeoffs for improving normal access latencies may have been right in the past but not for modern big disks. The seek time RPM have not improved very much in

Re: Anyone working on fsck?

2003-03-17 Thread Julian Elischer
On Mon, 17 Mar 2003, Bakul Shah wrote: Anyway, support for all of these have to be done in the filesystem first before fsck can benefit. yep If instead you spend time optimizing just fsck, you will likely make it far more complex (and potentially harder to get right). You talk like I

Re: Anyone working on fsck?

2003-03-17 Thread Bakul Shah
Now, before we go off and design YABFS, can we just get real for a second ? I leave it to others to design YAFS, I just wanted to complain about this one :-) Every few years I seriously look at speeding up fsck but give up. I remember even asking about it a few years ago on one of these

Re: Anyone working on fsck?

2003-03-17 Thread Bakul Shah
You talk like I have a choice :-) I cannot change ufs/ffs and even if I could the clients wouldn't go for it. What about changing the size of block size or cyl grp size? Do they change things much? The problem space is Fsck of UFS/FFS partitions is too slow for 200GB+ filesystems. The

Re: Anyone working on fsck?

2003-03-17 Thread Brooks Davis
On Mon, Mar 17, 2003 at 12:45:15PM -0800, Julian Elischer wrote: I might add that the test filesystem was 95% full with about 8,000,000 directories on it. It was populated with multiple copies of /bin and /etc as a test set :-) How much like you're real file mix does this look? If your real

Re: Anyone working on fsck?

2003-03-17 Thread Julian Elischer
On Mon, 17 Mar 2003, Bakul Shah wrote: Now, before we go off and design YABFS, can we just get real for a second ? I am skeptical you will get more than a factor of 2 improvement without changing the FS (but hey, that is 3 hours for Julian so I am sure he will be happy with that!).

Re: Anyone working on fsck?

2003-03-17 Thread Jeff Roberson
On Mon, 17 Mar 2003, Brooks Davis wrote: On Mon, Mar 17, 2003 at 12:45:15PM -0800, Julian Elischer wrote: I might add that the test filesystem was 95% full with about 8,000,000 directories on it. It was populated with multiple copies of /bin and /etc as a test set :-) How much like

Re: Anyone working on fsck?

2003-03-17 Thread Brad Knowles
At 10:39 PM +0100 2003/03/17, Poul-Henning Kamp wrote: Optimizing fsck is a valid project, I just wish it would be somebody who would also finish the last 30% who would do it. Just what are you saying? Is Julian Elischer not the right person to be working on this, because he has a history of

Re: Anyone working on fsck?

2003-03-17 Thread Greg 'groggy' Lehey
On Monday, 17 March 2003 at 22:39:02 +0100, Poul-Henning Kamp wrote: In message [EMAIL PROTECTED], Bakul Shah writes: UFS is the real problem here, not fsck. Its tradeoffs for improving normal access latencies may have been right in the past but not for modern big disks. The seek time RPM

Re: Anyone working on fsck?

2003-03-17 Thread Julian Elischer
On Mon, 17 Mar 2003, Brooks Davis wrote: On Mon, Mar 17, 2003 at 12:45:15PM -0800, Julian Elischer wrote: I might add that the test filesystem was 95% full with about 8,000,000 directories on it. It was populated with multiple copies of /bin and /etc as a test set :-) How much like

Re: Anyone working on fsck?

2003-03-17 Thread Terry Lambert
Bakul Shah wrote: UFS is the real problem here, not fsck. Its tradeoffs for improving normal access latencies may have been right in the past but not for modern big disks. The seek time RPM have not improved very much in the past 20 years while disk capacity has increased by a factor of

Re: Anyone working on fsck?

2003-03-17 Thread Terry Lambert
Julian Elischer wrote: The problem space is Fsck of UFS/FFS partitions is too slow for 200GB+ filesystems. The solution space can not contain any answer that includes redefining UFS/FFS. Welcome to the real world. :-) Use smaller than 200GB+ filesystems. 8-). -- Terry To Unsubscribe:

Re: Anyone working on fsck?

2003-03-17 Thread Terry Lambert
Bakul Shah wrote: I have been tending UNIX computers of all sorts for many years and there is one bit of wisdom that has yet to fail me: Every now and then, boot in single-user and run full fsck on all filesystems. If this had failed to be productive, I would have given

Re: Anyone working on fsck?

2003-03-17 Thread Terry Lambert
Jeff Roberson wrote: On Mon, 17 Mar 2003, Brooks Davis wrote: I am still intrested in improvements to fsck since I'm planning to buy several systems with two 1.4TB IDE RAID5 arrays in them soon. For these types of systems doing a block caching layer with a prefetch that understands how

Re: Anyone working on fsck?

2003-03-17 Thread Jeff Roberson
On Mon, 17 Mar 2003, Terry Lambert wrote: Jeff Roberson wrote: On Mon, 17 Mar 2003, Brooks Davis wrote: I am still intrested in improvements to fsck since I'm planning to buy several systems with two 1.4TB IDE RAID5 arrays in them soon. For these types of systems doing a block

Re: Anyone working on fsck?

2003-03-17 Thread Terry Lambert
Jeff Roberson wrote: On Mon, 17 Mar 2003, Terry Lambert wrote: Jeff Roberson wrote: On Mon, 17 Mar 2003, Brooks Davis wrote: I am still intrested in improvements to fsck since I'm planning to buy several systems with two 1.4TB IDE RAID5 arrays in them soon. For these types of

Re: Anyone working on fsck?

2003-03-17 Thread Kenneth D. Merry
On Mon, Mar 17, 2003 at 23:02:38 -0500, Jeff Roberson wrote: On Mon, 17 Mar 2003, Terry Lambert wrote: Jeff Roberson wrote: On Mon, 17 Mar 2003, Brooks Davis wrote: I am still intrested in improvements to fsck since I'm planning to buy several systems with two 1.4TB IDE RAID5 arrays

Re: Anyone working on fsck?

2003-03-17 Thread Bakul Shah
UFS is the real problem here, not fsck. Its tradeoffs for improving normal access latencies may have been right in the past but not for modern big disks. ... Sorry, but the track-to-track seek latency optimizations you are referring to are turned off, given the newfs defaults, and

Re: Anyone working on fsck?

2003-03-17 Thread Poul-Henning Kamp
In message [EMAIL PROTECTED], Brad Knowles writes: At 10:39 PM +0100 2003/03/17, Poul-Henning Kamp wrote: Optimizing fsck is a valid project, I just wish it would be somebody who would also finish the last 30% who would do it. Just what are you saying? Is Julian Elischer not the

Re: Anyone working on fsck?

2003-03-17 Thread Poul-Henning Kamp
In message [EMAIL PROTECTED], Greg 'groggy' Lehey writes: Optimizing fsck is a valid project, I just wish it would be somebody who would also finish the last 30% who would do it. Poul-Henning, how can you justify the second half of that sentence? I take exception to the implications. In

Re: Anyone working on fsck?

2003-03-17 Thread Terry Lambert
Bakul Shah wrote: Sorry, but the track-to-track seek latency optimizations you are referring to are turned off, given the newfs defaults, and have been for a very long time. I was thinking of the basic idea of cylinder groups as good for normal load, not so good for fsck when you have

Software RAID caching? (was: Anyone working on fsck?)

2003-03-17 Thread Greg 'groggy' Lehey
On Monday, 17 March 2003 at 23:02:38 -0500, Jeff Roberson wrote: On Mon, 17 Mar 2003, Terry Lambert wrote: Jeff Roberson wrote: On Mon, 17 Mar 2003, Brooks Davis wrote: I am still intrested in improvements to fsck since I'm planning to buy several systems with two 1.4TB IDE RAID5 arrays in

Re: Anyone working on fsck?

2003-03-17 Thread Mark Linimon
I'll stop as soon as KSE is finished, fair ? I'm very disappointed in this response. Poul, everything else I've read from you to date has been reasonable except for this posting. I would think that you, yourself, should be especially sensitive to criticism of unfinished projects. Things such

Re: Anyone working on fsck?

2003-03-17 Thread Peter Wemm
Poul-Henning Kamp wrote: In message [EMAIL PROTECTED], Greg 'groggy' Lehey writes: Optimizing fsck is a valid project, I just wish it would be somebody who would also finish the last 30% who would do it. Poul-Henning, how can you justify the second half of that sentence? I take