Re: [zfs-discuss] Scenario sanity check
First things first, the panic is a bug. Please file one with your OS supplier. More below... On Jul 6, 2012, at 4:55 PM, Ian Collins wrote: > On 07/ 7/12 11:29 AM, Brian Wilson wrote: >> On 07/ 6/12 04:17 PM, Ian Collins wrote: >>> On 07/ 7/12 08:34 AM, Brian Wilson wrote: Hello, I'd like a sanity check from people more knowledgeable than myself. I'm managing backups on a production system. Previously I was using another volume manager and filesystem on Solaris, and I've just switched to using ZFS. My model is - Production Server A Test Server B Mirrored storage arrays (HDS TruCopy if it matters) Backup software (TSM) Production server A sees the live volumes. Test Server B sees the TruCopy mirrors of the live volumes. (it sees the second storage array, the production server sees the primary array) Production server A shuts down zone C, and exports the zpools for zone C. Production server A splits the mirror to secondary storage array, leaving the mirror writable. Production server A re-imports the pools for zone C, and boots zone C. Test Server B imports the ZFS pool using -R /backup. Backup software backs up the mounted mirror volumes on Test Server B. Later in the day after the backups finish, a script exports the ZFS pools on test server B, and re-establishes the TruCopy mirror between the storage arrays. >>> That looks awfully complicated. Why don't you just clone a snapshot >>> and back up the clone? >>> >> Taking a snapshot and cloning incurs IO. Backing up the clone incurs a >> lot more IO reading off the disks and going over the network. These >> aren't acceptable costs in my situation. Yet it is acceptable to shut down the zones and export the pools? I'm interested to understand how a service outage is preferred over I/O? > So splitting a mirror and reconnecting it doesn't incur I/O? It does. >> The solution is complicated if you're starting from scratch. I'm >> working in an environment that already had all the pieces in place >> (offsite synchronous mirroring, a test server to mount stuff up on, >> scripts that automated the storage array mirror management, etc). It >> was setup that way specifically to accomplish short downtime outages for >> cold backups with minimal or no IO hit to production. So while it's >> complicated, when it was put together it was also the most obvious thing >> to do to drop my backup window to almost nothing, and keep all the IO >> from the backup from impacting production. And like I said, with a >> different volume manager, it's been rock solid for years. ... where data corruption is blissfully ignored? I'm not sure what volume manager you were using, but SVM has absolutely zero data integrity checking :-( And no, we do not miss using SVM :-) >> So, to ask the sanity check more specifically - >> Is it reasonable to expect ZFS pools to be exported, have their luns >> change underneath, then later import the same pool on those changed >> drives again? Yes, we do this quite frequently. And it is tested ad nauseum. Methinks it is simply a bug, perhaps one that is already fixed. > If you were splitting ZFS mirrors to read data from one half all would be > sweet (and you wouldn't have to export the pool). I guess the question here > is what does TruCopy do under the hood when you re-connect the mirror? Yes, this is one of the use cases for zpool split. However, zpool split creates a new pool, which is not what Brian wants, because to reattach the disks requires a full resilver. Using TrueCopy as he does, is a reasonable approach for Brian's use case. -- richard -- ZFS Performance and Training richard.ell...@richardelling.com +1-760-896-4422 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Scenario sanity check
On 07/ 7/12 11:29 AM, Brian Wilson wrote: On 07/ 6/12 04:17 PM, Ian Collins wrote: On 07/ 7/12 08:34 AM, Brian Wilson wrote: Hello, I'd like a sanity check from people more knowledgeable than myself. I'm managing backups on a production system. Previously I was using another volume manager and filesystem on Solaris, and I've just switched to using ZFS. My model is - Production Server A Test Server B Mirrored storage arrays (HDS TruCopy if it matters) Backup software (TSM) Production server A sees the live volumes. Test Server B sees the TruCopy mirrors of the live volumes. (it sees the second storage array, the production server sees the primary array) Production server A shuts down zone C, and exports the zpools for zone C. Production server A splits the mirror to secondary storage array, leaving the mirror writable. Production server A re-imports the pools for zone C, and boots zone C. Test Server B imports the ZFS pool using -R /backup. Backup software backs up the mounted mirror volumes on Test Server B. Later in the day after the backups finish, a script exports the ZFS pools on test server B, and re-establishes the TruCopy mirror between the storage arrays. That looks awfully complicated. Why don't you just clone a snapshot and back up the clone? Taking a snapshot and cloning incurs IO. Backing up the clone incurs a lot more IO reading off the disks and going over the network. These aren't acceptable costs in my situation. So splitting a mirror and reconnecting it doesn't incur I/O? The solution is complicated if you're starting from scratch. I'm working in an environment that already had all the pieces in place (offsite synchronous mirroring, a test server to mount stuff up on, scripts that automated the storage array mirror management, etc). It was setup that way specifically to accomplish short downtime outages for cold backups with minimal or no IO hit to production. So while it's complicated, when it was put together it was also the most obvious thing to do to drop my backup window to almost nothing, and keep all the IO from the backup from impacting production. And like I said, with a different volume manager, it's been rock solid for years. So, to ask the sanity check more specifically - Is it reasonable to expect ZFS pools to be exported, have their luns change underneath, then later import the same pool on those changed drives again? If you were splitting ZFS mirrors to read data from one half all would be sweet (and you wouldn't have to export the pool). I guess the question here is what does TruCopy do under the hood when you re-connect the mirror? -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Scenario sanity check
On 07/ 6/12 04:17 PM, Ian Collins wrote: On 07/ 7/12 08:34 AM, Brian Wilson wrote: Hello, I'd like a sanity check from people more knowledgeable than myself. I'm managing backups on a production system. Previously I was using another volume manager and filesystem on Solaris, and I've just switched to using ZFS. My model is - Production Server A Test Server B Mirrored storage arrays (HDS TruCopy if it matters) Backup software (TSM) Production server A sees the live volumes. Test Server B sees the TruCopy mirrors of the live volumes. (it sees the second storage array, the production server sees the primary array) Production server A shuts down zone C, and exports the zpools for zone C. Production server A splits the mirror to secondary storage array, leaving the mirror writable. Production server A re-imports the pools for zone C, and boots zone C. Test Server B imports the ZFS pool using -R /backup. Backup software backs up the mounted mirror volumes on Test Server B. Later in the day after the backups finish, a script exports the ZFS pools on test server B, and re-establishes the TruCopy mirror between the storage arrays. That looks awfully complicated. Why don't you just clone a snapshot and back up the clone? Taking a snapshot and cloning incurs IO. Backing up the clone incurs a lot more IO reading off the disks and going over the network. These aren't acceptable costs in my situation. The solution is complicated if you're starting from scratch. I'm working in an environment that already had all the pieces in place (offsite synchronous mirroring, a test server to mount stuff up on, scripts that automated the storage array mirror management, etc). It was setup that way specifically to accomplish short downtime outages for cold backups with minimal or no IO hit to production. So while it's complicated, when it was put together it was also the most obvious thing to do to drop my backup window to almost nothing, and keep all the IO from the backup from impacting production. And like I said, with a different volume manager, it's been rock solid for years. So, to ask the sanity check more specifically - Is it reasonable to expect ZFS pools to be exported, have their luns change underneath, then later import the same pool on those changed drives again? Thanks! Brian -- --- Brian Wilson, Solaris SE, UW-Madison DoIT Room 3114 CS&S608-263-8047 brian.wilson(a)doit.wisc.edu 'I try to save a life a day. Usually it's my own.' - John Crichton --- ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Scenario sanity check
On 07/ 7/12 08:34 AM, Brian Wilson wrote: Hello, I'd like a sanity check from people more knowledgeable than myself. I'm managing backups on a production system. Previously I was using another volume manager and filesystem on Solaris, and I've just switched to using ZFS. My model is - Production Server A Test Server B Mirrored storage arrays (HDS TruCopy if it matters) Backup software (TSM) Production server A sees the live volumes. Test Server B sees the TruCopy mirrors of the live volumes. (it sees the second storage array, the production server sees the primary array) Production server A shuts down zone C, and exports the zpools for zone C. Production server A splits the mirror to secondary storage array, leaving the mirror writable. Production server A re-imports the pools for zone C, and boots zone C. Test Server B imports the ZFS pool using -R /backup. Backup software backs up the mounted mirror volumes on Test Server B. Later in the day after the backups finish, a script exports the ZFS pools on test server B, and re-establishes the TruCopy mirror between the storage arrays. That looks awfully complicated. Why don't you just clone a snapshot and back up the clone? -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Cannot reset ZFS reservation and refreservation on volume
When creating a new zfs volume the calculated refreservation is greater than volsize to account for number of copies and metadata: root@test:~# zfs create -V 1G rpool/test root@test:~# zfs get -Hp volsize,volblocksize,copies,refreservation rpool/test rpool/test volsize 1073741824 local rpool/test volblocksize8192- rpool/test copies 1 default rpool/test refreservation 1107820544 local After I set refreservation to none, I am no longer able to reset refreservation back to the required refreservation, since there is a check in libzfs that prevents it: root@danstore2:/lib# zfs set refreservation=none rpool/test root@danstore2:/lib# zfs get -Hp volsize,volblocksize,copies,refreservation rpool/test rpool/test volsize 1073741824 local rpool/test volblocksize8192- rpool/test copies 1 default rpool/test refreservation 0 local root@danstore2:/lib# zfs set refreservation=1107820544 rpool/test cannot set property for 'rpool/test': 'refreservation' is greater than current volume size Is this an intended behavior or a bug? The same is true for reservation. Setting reservation on a volume is also limited to volsize, but reading the documentation (http://docs.oracle.com/cd/E19253-01/819-5461/gazvb/index.html) I understand reservation may be as large as the user wants it to be. I think this is so because: 1. "The quota and reservation properties are convenient for managing disk space consumed by datasets and their descendents" 2. " … descendents, such as snapshots and clones" If I understand correctly, the reservation on a volume accounts for all space consumed by the volume, its metadata and copies, and its descendant snapshots and clones, so it does not make any sense to limit it to volsize. Getting into libzfs code, I found that zfs_valid_proplist (in libzfs_dataset.c) specifically checks and prevents setting reservation and refreservation to more than volsize. I think the check should be removed for ZFS_PROP_RESERVATION, and limited to zvol_volsize_to_reservation(volsize, nvl) for ZFS_PROP_REFRESERVATION (when type == ZFS_TYPE_VOLUME). Dan PS: sorry if this message is a duplicate (I sent the original one from the wrong account). On 6 Jul 2012, at 0:00, Stefan Ring wrote: >> Actually, a write to memory for a memory mapped file is more similar to >> write(2). If two programs have the same file mapped then the effect on the >> memory they share is instantaneous because it is the same physical memory. >> A mmapped file becomes shared memory as soon as it is mapped at least twice. > > True, for some interpretation of "instantaneous". It does not > establish a happens-before relationship though, as > store-munmap/mmap-load does. > ___ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Scenario sanity check
Hello, I'd like a sanity check from people more knowledgeable than myself. I'm managing backups on a production system. Previously I was using another volume manager and filesystem on Solaris, and I've just switched to using ZFS. My model is - Production Server A Test Server B Mirrored storage arrays (HDS TruCopy if it matters) Backup software (TSM) Production server A sees the live volumes. Test Server B sees the TruCopy mirrors of the live volumes. (it sees the second storage array, the production server sees the primary array) Production server A shuts down zone C, and exports the zpools for zone C. Production server A splits the mirror to secondary storage array, leaving the mirror writable. Production server A re-imports the pools for zone C, and boots zone C. Test Server B imports the ZFS pool using -R /backup. Backup software backs up the mounted mirror volumes on Test Server B. Later in the day after the backups finish, a script exports the ZFS pools on test server B, and re-establishes the TruCopy mirror between the storage arrays. So.. I had this working fine with one zone on server A for a couple of months. This week I've added 6 more zones, each with two ZFS pools. The first night went okay. Last night, the test server B kernel panic'd well after the mirrored volumes zpools were imported, just after the TSM backup started reading all the ZFS pools to push it all to the enterprise backup environment. Here's the kernel panic message - Jul 6 03:04:55 riggs ^Mpanic[cpu22]/thread=2a10e81bca0: Jul 6 03:04:55 riggs unix: [ID 403854 kern.notice] assertion failed: 0 == dmu_buf_hold_array(os, object, offset, size, FALSE, FTAG, &numbufs, &dbp), file: ../../common/fs/zfs/dmu.c, line: 759 Jul 6 03:04:55 riggs unix: [ID 10 kern.notice] Jul 6 03:04:55 riggs genunix: [ID 723222 kern.notice] 02a10e81b4f0 genunix:assfail+74 (7af0f8c0, 7af0f910, 2f7, 190d000, 12a1800, 0) Jul 6 03:04:55 riggs genunix: [ID 179002 kern.notice] %l0-3: 0001 0001 0300f20fdf81 Jul 6 03:04:55 riggs %l4-7: 012a1800 01959400 Jul 6 03:04:55 riggs genunix: [ID 723222 kern.notice] 02a10e81b5a0 zfs:dmu_write+54 (300cbfd5c40, ad, a70, 20, 300b8c02800, 300f83414d0) Jul 6 03:04:55 riggs genunix: [ID 179002 kern.notice] %l0-3: 0038 0007 0194bd40 0194bc00 Jul 6 03:04:55 riggs %l4-7: 0001 030071bcb701 3006 3000 Jul 6 03:04:55 riggs genunix: [ID 723222 kern.notice] 02a10e81b670 zfs:space_map_sync+278 (3009babd130, b, 3009babcfe0, 20, 4, 58) Jul 6 03:04:55 riggs genunix: [ID 179002 kern.notice] %l0-3: 0020 0300b8c02800 0300b8c02820 0300b8c02858 Jul 6 03:04:55 riggs %l4-7: 7fff 7fff 22d9 0020 Jul 6 03:04:55 riggs genunix: [ID 723222 kern.notice] 02a10e81b760 zfs:metaslab_sync+2b0 (3009babcfc0, 1db7, 300f83414d0, 3009babd408, 300c9724000, 6003e24acc0) Jul 6 03:04:55 riggs genunix: [ID 179002 kern.notice] %l0-3: 0300cbfd5c40 03009babcff8 03009babd130 03009babd2d0 Jul 6 03:04:55 riggs %l4-7: 03009babcfe0 03009babd268 001a Jul 6 03:04:55 riggs genunix: [ID 723222 kern.notice] 02a10e81b820 zfs:vdev_sync+b8 (6003e24acc0, 1db7, 1db6, 3009babcfc0, 6003e24b000, 17) Jul 6 03:04:55 riggs genunix: [ID 179002 kern.notice] %l0-3: 0090 0012 06003e24acc0 0300c9724000 Jul 6 03:04:55 riggs %l4-7: 0009041ea000 Jul 6 03:04:55 riggs genunix: [ID 723222 kern.notice] 02a10e81b8d0 zfs:spa_sync+484 (300c9724000, 1db7, 3005fec09a8, 300c9724428, 1, 300cbfd5c40) Jul 6 03:04:55 riggs genunix: [ID 179002 kern.notice] %l0-3: 0300c9724280 030087c3e940 0300c6aae700 Jul 6 03:04:55 riggs %l4-7: 030080073520 0300c9724378 0300c9724300 0300c9724330 Jul 6 03:04:55 riggs genunix: [ID 723222 kern.notice] 02a10e81b9a0 zfs:txg_sync_thread+1b8 (30087c3e940, 183f9f0, 707a3130, 0, 2a10e81ba70, 0) Jul 6 03:04:55 riggs genunix: [ID 179002 kern.notice] %l0-3: 030087c3eb0e 030087c3eb08 030087c3eb0c Jul 6 03:04:55 riggs %l4-7: 1230fa07 030087c3eac8 030087c3ead0 1db7 Jul 6 03:04:55 riggs unix: [ID 10 kern.notice] So, I guess my question is - is what I'm doing sane? Or is there something inherint with ZFS that I'm missing that's going to cause this kernel panic to repeat? Best I can guess, it got upset when the pools were being read. I'm wondering of exporting the pools later in the day before re-syncing the SAN volumes to mirrors is causing weirdness. (because makes the mirrored volumes visible on Test Server B read-only until the split). I wouldn't think so, because the
Re: [zfs-discuss] Sol11 missing snapshot facility [solved]
-Original message- To: Carsten John ; CC: zfs-discuss@opensolaris.org; From: Ian Collins Sent: Thu 05-07-2012 21:40 Subject:Re: [zfs-discuss] Sol11 missing snapshot facility > On 07/ 5/12 11:32 PM, Carsten John wrote: > > -Original message- > > To: Carsten John; > > CC: zfs-discuss@opensolaris.org; > > From: Ian Collins > > Sent: Thu 05-07-2012 11:35 > > Subject:Re: [zfs-discuss] Sol11 missing snapshot facility > >> On 07/ 5/12 09:25 PM, Carsten John wrote: > >> > >>> Hi Ian, > >>> > >>> yes, I already checked that: > >>> > >>> svcs -a | grep zfs > >>> disabled 11:50:39 svc:/application/time-slider/plugin:zfs-send > >>> > >>> is the only service I get listed. > >>> > >> Odd. > >> > >> How did you install? > >> > >> Is the manifest there > >> (/lib/svc/manifest/system/filesystem/auto-snapshot.xml)? > >> > > Hi Ian, > > > > I installed from CD/DVD, but it might have been in a rush, as I needed to > replace a broken machine as quick as possible. > > > > The manifest is there: > > > > > > ls /lib/svc/manifest/system/filesystem/ > > . .. auto-snapshot.xml autofs.xml > local-fs.xml minimal-fs.xml rmvolmgr.xml root-fs.xml > ufs-quota.xml usr-fs.xml > > > > Running "svcadm restart manifest-import" should load it, or give you > some idea why it won't load. > > -- > Ian. > > Hi Ian, it did the trick, but I had to uninstall/install the time-slider package. thx for the help Carsten ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
