Re: [zfs-discuss] Unreliable ZFS backups or....
On Fri, Aug 11, 2006 at 05:25:11PM -0700, Peter Looyenga wrote: I looked into backing up ZFS and quite honostly I can't say I am convinced about its usefullness here when compared to the traditional ufsdump/restore. While snapshots are nice they can never substitute offline backups. And although you can keep quite some snapshots lying about it will consume diskspace, one of the reasons why people also keep offline backups. However, while you can make one using 'zfs send' it somewhat worries me that the only way to perform a restore is by restoring the entire filesystem (/snapshot). I somewhat shudder at the thought of having to restore /export/home this way to retrieve but a single file/directory. To achive that you can combine ZFS snapshots + ZFS send/receive. [ UFS ] - to restore one/two/... files use ufsrestore - to restore entire filesystem use ufsrestore [ ZFS ] - to restore one/two/... files use snapshot(s) - to restore entire filesystem use 'zfs receive' Am I overlooking something here or are people indeed resorting to tools like tar and the likes again to overcome all this? In my opinion ufsdump / ufsrestore was a major advantage over tar and I really would consider it a major drawback if that would be the only way to backup data in such a way where it can be more easily restored. Now in zfs-era :-) it seems we have to change way of thinking about filesystems. Usually system administrators are restoring two things: - particular files - entire filesystems Both are achievable by above combination. What is more important is that zfs-way (restore a file) seems to be easier. :-) przemol ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Re: 3510 HW RAID vs 3510 JBOD ZFS SOFTWARE RAID
The test case was build 38, Solaris 11, a 2 GB file, initially created with 1 MB SW, and a recsize of 8 KB, on a pool with two raid-z 5+1, accessed with 24 threads of 8 KB RW, for 500,000 ops or 40 seconds which ever came first. The result at the pool level was 78% of the operations were RR, all overhead. Hi David, Could this bug (now fixed) have hit you ? 6424554 full block re-writes need not read data in -r ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Significant pauses during zfs writes
Hi Bob, Looks like : 6415647 Sequential writing is jumping http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6415647 -r Roch BourbonnaisSun Microsystems, Icnc-Grenoble Senior Performance Analyst 180, Avenue De L'Europe, 38330, Montbonnot Saint Martin, France Performance Availability Engineering http://icncweb.france/~rbourbon http://blogs.sun.com/roller/page/roch [EMAIL PROTECTED] (+33).4.76.18.83.20 Bob Evans writes: HI, Just getting my feet wet with zfs. I set up a test system (Sunblade 1000, dual channel scsi card, disk array with 14x18GB 15K RPM SCSI disks) and was trying to write a large file (10 GB) to the array to see how it performed. I configured the raid using raidz. During the write, I saw the disk access lights come on, but I noticed a peculiar behavior. The system would write to the disk, but then pause for a few seconds, then contineu, then pause for a few seconds. I saw the same behavior when I made a smaller raidz using 4x36 GB scsi drives in a different enclosure. Since I'm new to zfs, and realize that I'm probably missing something, I was hoping somebody might help shed some light on my problem. Thanks! This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Significant pauses during zfs writes
Yes James is right this is normal behaviour. Unless the writes are synchronous (O_DSYNC) or explicitely flushed (fsync()) then they are batched up, written out and committed as a transaction every txg_time (5 seconds). Neil. James C. McPherson wrote: Bob Evans wrote: Just getting my feet wet with zfs. I set up a test system (Sunblade 1000, dual channel scsi card, disk array with 14x18GB 15K RPM SCSI disks) and was trying to write a large file (10 GB) to the array to see how it performed. I configured the raid using raidz. During the write, I saw the disk access lights come on, but I noticed a peculiar behavior. The system would write to the disk, but then pause for a few seconds, then contineu, then pause for a few seconds. I saw the same behavior when I made a smaller raidz using 4x36 GB scsi drives in a different enclosure. Since I'm new to zfs, and realize that I'm probably missing something, I was hoping somebody might help shed some light on my problem. Hi Bob, I'm pretty sure that's not a problem that you're seeing, just ZFS' normal behaviour. Writes are coalesced as much as possible, so the pauses that you observed are most likely going to be the system waiting for suitable IOs to be gathered up and sent out to your storage. If you want to examine this a bit more then might I suggest the DTrace Toolkit's iosnoop utility. best regards, James C. McPherson ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- Neil ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Significant pauses during zfs writes
Neil Perrin writes: Yes James is right this is normal behaviour. Unless the writes are synchronous (O_DSYNC) or explicitely flushed (fsync()) then they are batched up, written out and committed as a transaction every txg_time (5 seconds). Neil. James C. McPherson wrote: Bob Evans wrote: Just getting my feet wet with zfs. I set up a test system (Sunblade 1000, dual channel scsi card, disk array with 14x18GB 15K RPM SCSI disks) and was trying to write a large file (10 GB) to the array to see how it performed. I configured the raid using raidz. During the write, I saw the disk access lights come on, but I noticed a peculiar behavior. The system would write to the disk, but then pause for a few seconds, then contineu, then pause for a few seconds. I saw the same behavior when I made a smaller raidz using 4x36 GB scsi drives in a different enclosure. Since I'm new to zfs, and realize that I'm probably missing something, I was hoping somebody might help shed some light on my problem. Hi Bob, I'm pretty sure that's not a problem that you're seeing, just ZFS' normal behaviour. Writes are coalesced as much as possible, so the pauses that you observed are most likely going to be the system waiting for suitable IOs to be gathered up and sent out to your storage. If you want to examine this a bit more then might I suggest the DTrace Toolkit's iosnoop utility. best regards, James C. McPherson ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- Neil ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss I also would agree if the application is burning 100% of a CPU. But if the application is being throttled at times _and_the storage is itself not 100% exercised then I believe something is wrong and we have that anomalous jumpyness. So Bob, is the application burning a full CPU ? -r ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Re: Lots of seeks?
Incidentally, this is part of how QFS gets its performance for streaming I/O. We use an allocate forward policy, allow very largeallocation blocks, and separate the metadata from data. This allows us to write (or read) data in fairly large I/O requests, without unnecessary disk head motion. I believe ZFS allocate forward both data and metadata but I don't think this causes unnecessary disk head motion. -r ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Re: Significant pauses during zfs writes
I'm starting simple, there is no app. I have a 10GB file (called foo) on the internal FC drive, I did a zfs create raidz bar then ran cp foo /bar/, so there is no cpu activity due to an app. As a test case, this took 7 min 30 sec to copy to the zfs partition. I removed the pool, formatted a single drive, and then tried to copy the same file to a single drive that was nfs mounted. This only took 5 min 30 sec. Watching disk activity, it seems like I get 2-4 seconds of writes, followed by 2-4 seconds of inactivity. Bob This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Re: Significant pauses during zfs writes
As added information, top reports that cp is using about 25% of the single cpu. There are no other apps running. Bob This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Re: Significant pauses during zfs writes
Bob Evans writes: I'm starting simple, there is no app. I have a 10GB file (called foo) on the internal FC drive, I did a zfs create raidz bar then ran cp foo /bar/, so there is no cpu activity due to an app. As a test case, this took 7 min 30 sec to copy to the zfs partition. I removed the pool, formatted a single drive, and then tried to copy the same file to a single drive that was nfs mounted. This only took 5 min 30 sec. Watching disk activity, it seems like I get 2-4 seconds of writes, followed by 2-4 seconds of inactivity. Bob That's points to the jumpiness issue. BTW, cp(1) is your application ;-) If it's not 100% on CPU then it's not your throttle, the storage is and _that_ should be busy 100% of the time. -r This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Re: Significant pauses during zfs writes
One last tidbit, for what it is worth. Rather than watch top, I ran xcpustate. It seems that just as the writes pause, the cpu looks like it hits 100% (or very close), then it falls back down to its lower level. I'm still getting used to Solaris 10 as well, so if you have a DTrace script you'd recommend me running, please feel free to suggest it. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Re: Significant pauses during zfs writes
Bob Evans writes: One last tidbit, for what it is worth. Rather than watch top, I ran xcpustate. It seems that just as the writes pause, the cpu looks like it hits 100% (or very close), then it falls back down to its lower level. I'm still getting used to Solaris 10 as well, so if you have a DTrace script you'd recommend me running, please feel free to suggest it. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss I had overlooked that this could be a single CPU system. For such a test the potential limiting factors can be either of: 1. storage. 2. the single cp thread burning a CPU. 3. total system CPU overload. You've already indicated that storage is not 100% busy. The cp(1) thread is not burning a CPU. And it looks like the system is not CPU saturated, right ? I'd argue that the conjunction of those 3 things somewhat points to 'a problem'. -r ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Re: 3510 HW RAID vs 3510 JBOD ZFS SOFTWARE RAID
Robert Milkowski wrote: ps. however I'm really concerned with ZFS behavior when a pool is almost full, there're lot of write transactions to that pool and server is restarted forcibly or panics. I observed that file systems on that pool will mount in 10-30 minutes each during zfs mount -a, and one CPU is completely consumed. It's during system start-up so basically whole system boots waits for it. It means additional 1 hour downtime. This is something really unexpected for me and unfortunately no one was really interested in my report - I know people are busy. But still if it hits other users when zfs pools will be already populated people won't be happy. For more details see my post here with subject: zfs mount stuck in zil_replay. That problem must have fallen through the cracks. Yes we are busy, but we really do care about your experiences and bugs. I have just raised a bug to cover this issue: 6460107 Extremely slow mounts after panic - searching space maps during replay Thanks for reporting this and helping make ZFS better. Neil ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Significant pauses during zfs writes
Hey, Bob - It might be worth exploring where your data stream for the writes was coming from. Moreover, it might be worth exploring how fast it was filling up caches for writing. Were you delivering enough data to keep the disks busy 100% of the time? I have been tricked by this before... :) Nathan. On Tue, 2006-08-15 at 01:38, James C. McPherson wrote: Bob Evans wrote: Just getting my feet wet with zfs. I set up a test system (Sunblade 1000, dual channel scsi card, disk array with 14x18GB 15K RPM SCSI disks) and was trying to write a large file (10 GB) to the array to see how it performed. I configured the raid using raidz. During the write, I saw the disk access lights come on, but I noticed a peculiar behavior. The system would write to the disk, but then pause for a few seconds, then contineu, then pause for a few seconds. I saw the same behavior when I made a smaller raidz using 4x36 GB scsi drives in a different enclosure. Since I'm new to zfs, and realize that I'm probably missing something, I was hoping somebody might help shed some light on my problem. Hi Bob, I'm pretty sure that's not a problem that you're seeing, just ZFS' normal behaviour. Writes are coalesced as much as possible, so the pauses that you observed are most likely going to be the system waiting for suitable IOs to be gathered up and sent out to your storage. If you want to examine this a bit more then might I suggest the DTrace Toolkit's iosnoop utility. best regards, James C. McPherson ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Proposal: zfs create -o
Eric Schrock wrote: This case adds a new option, 'zfs create -o', which allows for any ZFS property to be set at creation time. Multiple '-o' options can appear in the same subcommand. Specifying the same property multiple times in the same command results in an error. For example: # zfs create -o compression=on -o mountpoint=/export pool/fs Looks fine to me. The one minor nit I have is that this even more shows up the positional parameters problem that zfs create and other zfs(1) commands have. Some people don't mind it and I've created interfaces like it for years; recently however I've been trying to avoid doing so, or at least giving an alternative. For this case would it be possible to have this: # zfs create -o compression=on -o mountpoint=/export -o name=pool/fs The zfs list command already allows -o name (and zfs get should but doesn't). I don't feel that strongly about this but thing it might be helpful in some cases. -- Darren J Moffat ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] user quotas vs filesystem quotas?
Jeff A. Earickson wrote: Are there any plans/schemes for per-user quotas within a ZFS filesystem, akin to the UFS quotaon(1M) mechanism? I take it that quotaon won't work with a ZFS filesystem, right? Suggestions please? My notion right now is to drop quotas for /var/mail. An alternative might be to switch to using mail server software that does this internally. For example the Messaging Server software that Sun produces has this functionality. Having the mail quota in the mail server allows you to do much more interesting things, especially if you have single instance storage for attachments supported by your mail server. It also often gives you auto mail expiry and clean up on the server side. -- Darren J Moffat ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
