Re: [zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
From: Richard Elling [mailto:richard.ell...@gmail.com] Pedantically, a pool can be made in a file, so it works the same... Pool can only be made in a file, by a system that is able to create a pool. Point is, his receiving system runs linux and doesn't have any zfs; his receiving system is remote from his sending system, and it has been suggested that he might consider making an iscsi target available, so the sending system could zpool create and zfs receive directly into a file or device on the receiving system, but it doesn't seem as if that's going to be possible for him - he's expecting to transport the data over ssh. So he's looking for a way to do a zfs receive on a linux system, transported over ssh. Suggested answers so far include building a VM on the receiving side, to run openindiana (or whatever) or using zfs-fuse-linux. He is currently writing his zfs send datastream into a series of files on the receiving system, but this has a few disadvantages as compared to doing zfs receive on the receiving side. Namely, increased risk of data loss and less granularity for restores. For these reasons, it's been suggested to find a way of receiving via zfs receive and he's exploring the possibilities of how to improve upon this situation. Namely, how to zfs receive on a remote linux system via ssh, instead of cat'ing or redirecting into a series of files. There, I think I've recapped the whole thread now. ;-) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
2012-10-12 16:50, Edward Ned Harvey (opensolarisisdeadlongliveopensolaris) пишет: So he's looking for a way to do a zfs receive on a linux system, transported over ssh. Suggested answers so far include building a VM on the receiving side, to run openindiana (or whatever) or using zfs-fuse-linux. For completeness, if iSCSI target on the receiving host or another similar solution is implemented, the secure networking part of zfs send over ssh (local sending into a pool on an iSCSI target) can be done by a VPN, i.e. OpenVPN which uses the same OpenSSL encryption. //Jim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
Jim, I'm trying to contact you off-list, but it doesn't seem to be working. Can you please contact me off-list? ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
On Oct 12, 2012, at 5:50 AM, Edward Ned Harvey (opensolarisisdeadlongliveopensolaris) opensolarisisdeadlongliveopensola...@nedharvey.com wrote: From: Richard Elling [mailto:richard.ell...@gmail.com] Pedantically, a pool can be made in a file, so it works the same... Pool can only be made in a file, by a system that is able to create a pool. You can't send a pool, you can only send a dataset. Whether you receive the dataset into a pool or file is a minor nit, the send stream itself is consistent. Point is, his receiving system runs linux and doesn't have any zfs; his receiving system is remote from his sending system, and it has been suggested that he might consider making an iscsi target available, so the sending system could zpool create and zfs receive directly into a file or device on the receiving system, but it doesn't seem as if that's going to be possible for him - he's expecting to transport the data over ssh. So he's looking for a way to do a zfs receive on a linux system, transported over ssh. Suggested answers so far include building a VM on the receiving side, to run openindiana (or whatever) or using zfs-fuse-linux. He is currently writing his zfs send datastream into a series of files on the receiving system, but this has a few disadvantages as compared to doing zfs receive on the receiving side. Namely, increased risk of data loss and less granularity for restores. For these reasons, it's been suggested to find a way of receiving via zfs receive and he's exploring the possibilities of how to improve upon this situation. Namely, how to zfs receive on a remote linux system via ssh, instead of cat'ing or redirecting into a series of files. There, I think I've recapped the whole thread now. ;-) Yep, and cat works fine. -- richard -- richard.ell...@richardelling.com +1-760-896-4422 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
From: Richard Elling [mailto:richard.ell...@gmail.com] Read it again he asked, On that note, is there a minimal user-mode zfs thing that would allow receiving a stream into an image file? Something like: zfs send ... | ssh user@host cat file He didn't say he wanted to cat to a file. But it doesn't matter. It was only clear from context, responding to the advice of zfs receiveing into a zpool-in-a-file, that it was clear he was asking about doing a zfs receive into a file, not just cat. If you weren't paying close attention to the thread, it would be easy to misunderstand what he was asking for. When he asked for minimal user-mode he meant, something less than a full-blown OS installation just for the purpose of zfs receive. He went on to say, he was considering zfs-fuse-on-linux. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
On Oct 11, 2012, at 6:03 AM, Edward Ned Harvey (opensolarisisdeadlongliveopensolaris) opensolarisisdeadlongliveopensola...@nedharvey.com wrote: From: Richard Elling [mailto:richard.ell...@gmail.com] Read it again he asked, On that note, is there a minimal user-mode zfs thing that would allow receiving a stream into an image file? Something like: zfs send ... | ssh user@host cat file He didn't say he wanted to cat to a file. But it doesn't matter. It was only clear from context, responding to the advice of zfs receiveing into a zpool-in-a-file, that it was clear he was asking about doing a zfs receive into a file, not just cat. If you weren't paying close attention to the thread, it would be easy to misunderstand what he was asking for. Pedantically, a pool can be made in a file, so it works the same... When he asked for minimal user-mode he meant, something less than a full-blown OS installation just for the purpose of zfs receive. He went on to say, he was considering zfs-fuse-on-linux. ... though I'm not convinced zfs-fuse supports files, whereas illumos/Solaris does. Perhaps a linux fuse person can respond. -- richard -- richard.ell...@richardelling.com +1-760-896-4422 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Richard Elling If the recipient system doesn't support zfs receive, [...] On that note, is there a minimal user-mode zfs thing that would allow receiving a stream into an image file? No need for file/directory access etc. cat :-) He was asking if it's possible to do zfs receive on a system that doesn't natively support zfs. The answer is no, unless you want to consider fuse or similar. I can't speak about zfs on fuse or anything - except that I personally wouldn't trust it. There are differences even between zfs on solaris versus freebsd, vs whatever, all of which are fully supported, much better than zfs on fuse. But different people use and swear by all of these things - so maybe it would actually be a good solution for you. The direction I would personally go would be an openindiana virtual machine to do the zfs receive. I was thinking maybe the zfs-fuse-on-linux project may have suitable bits? I'm sure most Linux distros have cat hehe. Anyway. Answered above. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
On Oct 10, 2012, at 9:29 AM, Edward Ned Harvey (opensolarisisdeadlongliveopensolaris) opensolarisisdeadlongliveopensola...@nedharvey.com wrote: From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Richard Elling If the recipient system doesn't support zfs receive, [...] On that note, is there a minimal user-mode zfs thing that would allow receiving a stream into an image file? No need for file/directory access etc. cat :-) He was asking if it's possible to do zfs receive on a system that doesn't natively support zfs. The answer is no, unless you want to consider fuse or similar. Read it again he asked, On that note, is there a minimal user-mode zfs thing that would allow receiving a stream into an image file? Something like: zfs send ... | ssh user@host cat file I can't speak about zfs on fuse or anything - except that I personally wouldn't trust it. There are differences even between zfs on solaris versus freebsd, vs whatever, all of which are fully supported, much better than zfs on fuse. But different people use and swear by all of these things - so maybe it would actually be a good solution for you. The direction I would personally go would be an openindiana virtual machine to do the zfs receive. I was thinking maybe the zfs-fuse-on-linux project may have suitable bits? I'm sure most Linux distros have cat hehe. Anyway. Answered above. -- richard -- richard.ell...@richardelling.com +1-760-896-4422 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
Ok, so, after reading a bit more of this discussion and after playing around at the weekend, i have a couple of questions to ask... 1: Do my pools need to be the same? for example, the pool in the datacenter is 2 1Tb drives in Mirror. in house i have 5 200Gb virtual drives in RAIDZ1, giving 800Gb usable. If i am backing up stuff to the home server, can i still do a ZFS Send, even though underlying system is different? 2: If i give out a partition as an iSCSI LUN, can this be ZFS Sended as normal, or is there any difference? Thanks. --Tiernan On Mon, Oct 8, 2012 at 3:51 AM, Richard Elling richard.ell...@gmail.comwrote: On Oct 7, 2012, at 3:50 PM, Johannes Totz johan...@jo-t.de wrote: On 05/10/2012 15:01, Edward Ned Harvey (opensolarisisdeadlongliveopensolaris) wrote: From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Tiernan OToole I am in the process of planning a system which will have 2 ZFS servers, one on site, one off site. The on site server will be used by workstations and servers in house, and most of that will stay in house. There will, however, be data i want backed up somewhere else, which is where the offsite server comes in... This server will be sitting in a Data Center and will have some storage available to it (the whole server currently has 2 3Tb drives, though they are not dedicated to the ZFS box, they are on VMware ESXi). There is then some storage (currently 100Gb, but more can be requested) of SFTP enabled backup which i plan to use for some snapshots, but more on that later. Anyway, i want to confirm my plan and make sure i am not missing anything here... * build server in house with storage, pools, etc... * have a server in data center with enough storage for its reason, plus the extra for offsite backup * have one pool set as my offsite pool... anything in here should be backed up off site also... * possibly have another set as very offsite which will also be pushed to the SFTP server, but not sure... * give these pools out via SMB/NFS/iSCSI * every 6 or so hours take a snapshot of the 2 offsite pools. * do a ZFS send to the data center box * nightly, on the very offsite pool, do a ZFS send to the SFTP server * if anything goes wrong (my server dies, DC server dies, etc), Panic, download, pray... the usual... :) Anyway, I want to make sure i am doing this correctly... Is there anything on that list that sounds stupid or am i doing anything wrong? am i missing anything? Also, as a follow up question, but slightly unrelated, when it comes to the ZFS Send, i could use SSH to do the send, directly to the machine... Or i could upload the compressed, and possibly encrypted dump to the server... Which, for resume-ability and speed, would be suggested? And if i where to go with an upload option, any suggestions on what i should use? It is recommended, whenever possible, you should pipe the zfs send directly into a zfs receive on the receiving system. For two solid reasons: If a single bit is corrupted, the whole stream checksum is wrong and therefore the whole stream is rejected. So if this occurs, you want to detect it (in the form of one incremental failed) and then correct it (in the form of the next incremental succeeding). Whereas, if you store your streams on storage, it will go undetected, and everything after that point will be broken. If you need to do a restore, from a stream stored on storage, then your only choice is to restore the whole stream. You cannot look inside and just get one file. But if you had been doing send | receive, then you obviously can look inside the receiving filesystem and extract some individual specifics. If the recipient system doesn't support zfs receive, [...] On that note, is there a minimal user-mode zfs thing that would allow receiving a stream into an image file? No need for file/directory access etc. cat :-) I was thinking maybe the zfs-fuse-on-linux project may have suitable bits? I'm sure most Linux distros have cat -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- Tiernan O'Toole blog.lotas-smartman.net www.geekphotographer.com www.tiernanotoole.ie ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
On 10/08/12 20:08, Tiernan OToole wrote: Ok, so, after reading a bit more of this discussion and after playing around at the weekend, i have a couple of questions to ask... 1: Do my pools need to be the same? for example, the pool in the datacenter is 2 1Tb drives in Mirror. in house i have 5 200Gb virtual drives in RAIDZ1, giving 800Gb usable. If i am backing up stuff to the home server, can i still do a ZFS Send, even though underlying system is different? Yes you can, just make sure you have enough space! 2: If i give out a partition as an iSCSI LUN, can this be ZFS Sended as normal, or is there any difference? It can be sent as normal. -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
Cool beans lads. Thanks! On Mon, Oct 8, 2012 at 8:17 AM, Ian Collins i...@ianshome.com wrote: On 10/08/12 20:08, Tiernan OToole wrote: Ok, so, after reading a bit more of this discussion and after playing around at the weekend, i have a couple of questions to ask... 1: Do my pools need to be the same? for example, the pool in the datacenter is 2 1Tb drives in Mirror. in house i have 5 200Gb virtual drives in RAIDZ1, giving 800Gb usable. If i am backing up stuff to the home server, can i still do a ZFS Send, even though underlying system is different? Yes you can, just make sure you have enough space! 2: If i give out a partition as an iSCSI LUN, can this be ZFS Sended as normal, or is there any difference? It can be sent as normal. -- Ian. __**_ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/**mailman/listinfo/zfs-discusshttp://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- Tiernan O'Toole blog.lotas-smartman.net www.geekphotographer.com www.tiernanotoole.ie ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
On 05/10/2012 15:01, Edward Ned Harvey (opensolarisisdeadlongliveopensolaris) wrote: From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Tiernan OToole I am in the process of planning a system which will have 2 ZFS servers, one on site, one off site. The on site server will be used by workstations and servers in house, and most of that will stay in house. There will, however, be data i want backed up somewhere else, which is where the offsite server comes in... This server will be sitting in a Data Center and will have some storage available to it (the whole server currently has 2 3Tb drives, though they are not dedicated to the ZFS box, they are on VMware ESXi). There is then some storage (currently 100Gb, but more can be requested) of SFTP enabled backup which i plan to use for some snapshots, but more on that later. Anyway, i want to confirm my plan and make sure i am not missing anything here... * build server in house with storage, pools, etc... * have a server in data center with enough storage for its reason, plus the extra for offsite backup * have one pool set as my offsite pool... anything in here should be backed up off site also... * possibly have another set as very offsite which will also be pushed to the SFTP server, but not sure... * give these pools out via SMB/NFS/iSCSI * every 6 or so hours take a snapshot of the 2 offsite pools. * do a ZFS send to the data center box * nightly, on the very offsite pool, do a ZFS send to the SFTP server * if anything goes wrong (my server dies, DC server dies, etc), Panic, download, pray... the usual... :) Anyway, I want to make sure i am doing this correctly... Is there anything on that list that sounds stupid or am i doing anything wrong? am i missing anything? Also, as a follow up question, but slightly unrelated, when it comes to the ZFS Send, i could use SSH to do the send, directly to the machine... Or i could upload the compressed, and possibly encrypted dump to the server... Which, for resume-ability and speed, would be suggested? And if i where to go with an upload option, any suggestions on what i should use? It is recommended, whenever possible, you should pipe the zfs send directly into a zfs receive on the receiving system. For two solid reasons: If a single bit is corrupted, the whole stream checksum is wrong and therefore the whole stream is rejected. So if this occurs, you want to detect it (in the form of one incremental failed) and then correct it (in the form of the next incremental succeeding). Whereas, if you store your streams on storage, it will go undetected, and everything after that point will be broken. If you need to do a restore, from a stream stored on storage, then your only choice is to restore the whole stream. You cannot look inside and just get one file. But if you had been doing send | receive, then you obviously can look inside the receiving filesystem and extract some individual specifics. If the recipient system doesn't support zfs receive, [...] On that note, is there a minimal user-mode zfs thing that would allow receiving a stream into an image file? No need for file/directory access etc. I was thinking maybe the zfs-fuse-on-linux project may have suitable bits? ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
On Oct 7, 2012, at 3:50 PM, Johannes Totz johan...@jo-t.de wrote: On 05/10/2012 15:01, Edward Ned Harvey (opensolarisisdeadlongliveopensolaris) wrote: From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Tiernan OToole I am in the process of planning a system which will have 2 ZFS servers, one on site, one off site. The on site server will be used by workstations and servers in house, and most of that will stay in house. There will, however, be data i want backed up somewhere else, which is where the offsite server comes in... This server will be sitting in a Data Center and will have some storage available to it (the whole server currently has 2 3Tb drives, though they are not dedicated to the ZFS box, they are on VMware ESXi). There is then some storage (currently 100Gb, but more can be requested) of SFTP enabled backup which i plan to use for some snapshots, but more on that later. Anyway, i want to confirm my plan and make sure i am not missing anything here... * build server in house with storage, pools, etc... * have a server in data center with enough storage for its reason, plus the extra for offsite backup * have one pool set as my offsite pool... anything in here should be backed up off site also... * possibly have another set as very offsite which will also be pushed to the SFTP server, but not sure... * give these pools out via SMB/NFS/iSCSI * every 6 or so hours take a snapshot of the 2 offsite pools. * do a ZFS send to the data center box * nightly, on the very offsite pool, do a ZFS send to the SFTP server * if anything goes wrong (my server dies, DC server dies, etc), Panic, download, pray... the usual... :) Anyway, I want to make sure i am doing this correctly... Is there anything on that list that sounds stupid or am i doing anything wrong? am i missing anything? Also, as a follow up question, but slightly unrelated, when it comes to the ZFS Send, i could use SSH to do the send, directly to the machine... Or i could upload the compressed, and possibly encrypted dump to the server... Which, for resume-ability and speed, would be suggested? And if i where to go with an upload option, any suggestions on what i should use? It is recommended, whenever possible, you should pipe the zfs send directly into a zfs receive on the receiving system. For two solid reasons: If a single bit is corrupted, the whole stream checksum is wrong and therefore the whole stream is rejected. So if this occurs, you want to detect it (in the form of one incremental failed) and then correct it (in the form of the next incremental succeeding). Whereas, if you store your streams on storage, it will go undetected, and everything after that point will be broken. If you need to do a restore, from a stream stored on storage, then your only choice is to restore the whole stream. You cannot look inside and just get one file. But if you had been doing send | receive, then you obviously can look inside the receiving filesystem and extract some individual specifics. If the recipient system doesn't support zfs receive, [...] On that note, is there a minimal user-mode zfs thing that would allow receiving a stream into an image file? No need for file/directory access etc. cat :-) I was thinking maybe the zfs-fuse-on-linux project may have suitable bits? I'm sure most Linux distros have cat -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
Good morning. I am in the process of planning a system which will have 2 ZFS servers, one on site, one off site. The on site server will be used by workstations and servers in house, and most of that will stay in house. There will, however, be data i want backed up somewhere else, which is where the offsite server comes in... This server will be sitting in a Data Center and will have some storage available to it (the whole server currently has 2 3Tb drives, though they are not dedicated to the ZFS box, they are on VMware ESXi). There is then some storage (currently 100Gb, but more can be requested) of SFTP enabled backup which i plan to use for some snapshots, but more on that later. Anyway, i want to confirm my plan and make sure i am not missing anything here... * build server in house with storage, pools, etc... * have a server in data center with enough storage for its reason, plus the extra for offsite backup * have one pool set as my offsite pool... anything in here should be backed up off site also... * possibly have another set as very offsite which will also be pushed to the SFTP server, but not sure... * give these pools out via SMB/NFS/iSCSI * every 6 or so hours take a snapshot of the 2 offsite pools. * do a ZFS send to the data center box * nightly, on the very offsite pool, do a ZFS send to the SFTP server * if anything goes wrong (my server dies, DC server dies, etc), Panic, download, pray... the usual... :) Anyway, I want to make sure i am doing this correctly... Is there anything on that list that sounds stupid or am i doing anything wrong? am i missing anything? Also, as a follow up question, but slightly unrelated, when it comes to the ZFS Send, i could use SSH to do the send, directly to the machine... Or i could upload the compressed, and possibly encrypted dump to the server... Which, for resume-ability and speed, would be suggested? And if i where to go with an upload option, any suggestions on what i should use? Thanks. -- Tiernan O'Toole blog.lotas-smartman.net www.geekphotographer.com www.tiernanotoole.ie ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
2012-10-05 11:17, Tiernan OToole wrote: Also, as a follow up question, but slightly unrelated, when it comes to the ZFS Send, i could use SSH to do the send, directly to the machine... Or i could upload the compressed, and possibly encrypted dump to the server... Which, for resume-ability and speed, would be suggested? And if i where to go with an upload option, any suggestions on what i should use? As for this, the answer depends on network bandwidth, reliability, and snapshot file size - ultimately, on the probability and retry cost of an error during transmission. Many posters on the list strongly object to using files as storage for snapshot streams, because in reliability this is (may be) worse than a single-disk pool and bitrot on it - a single-bit error in a snapshot file can render it and all newer snapshots invalid and un-importable. Still, given enough scratch space on the sending and receiving sides and a bad (slow, glitchy) network in-between, I did go with compressed files of zfs-send streams (perhaps making recursion myself and using smaller files of one snapshot each - YMMV). For compression on multiCPU senders I can strongly suggest pigz --fast $filename (I did have problems in pigz-1.7.1 compressing several files with one command, maybe that's fixed now). If you're tight on space/transfer size more than on CPU, you can try other parallel algos - pbzip2, p7zip, etc. Likewise, you can also pass the file into an encryptor of your choice. Then I can rsync these files to the receiving server, using rsync -c and/or md5sum, sha256sum, sha1sum or whatever tool(s) of your liking to validate that the files received match those sent - better safe than sorry. I'm usually using rsync -cavPHK for any work, which gives you retryable transfers in case network goes down, status bar, directory recursion and hardlink support among other things. NFS is also retryable if so configured (even if the receiver gets rebooted in the process), and if you, for example, already have VPN between two sites, you might find it faster than rsync which involves extra encryption - maybe redundant in VPN case. When the scratch area on the receiver has got and validated the compressed snapshot stream, I can gzcat it and pipe into zfs recv. This ultimately validates that the zfs-send stream arrived intact and is fully receivable, and only then I can delete the temporary files involved - or retry the send from different steps (it is possible that the initial file was corrupted in RAM, etc.) Note that such approach via files essentially disables zfs-send deduplication which may be available in protocol between two active zfs commands, but AFAIK this does not preclude you from receiving data into deduped datasets - local dedup happens upon block writes anyway, like compression, encryption and stuff like that. HTH, //Jim Klimov ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
Thanks for that Jim! Sounds like a plan there... One question about the storing ZFS dumps in a file... So, the idea of storing the data in a SFTP server which has an unknown underlying file system... Is that defiantly off limits, or can it be done? and should i be doing a full dump or just an incremental one? maybe incremental daily, and then a full dump weekly? Thanks! --Tiernan On Fri, Oct 5, 2012 at 9:36 AM, Jim Klimov jimkli...@cos.ru wrote: 2012-10-05 11:17, Tiernan OToole wrote: Also, as a follow up question, but slightly unrelated, when it comes to the ZFS Send, i could use SSH to do the send, directly to the machine... Or i could upload the compressed, and possibly encrypted dump to the server... Which, for resume-ability and speed, would be suggested? And if i where to go with an upload option, any suggestions on what i should use? As for this, the answer depends on network bandwidth, reliability, and snapshot file size - ultimately, on the probability and retry cost of an error during transmission. Many posters on the list strongly object to using files as storage for snapshot streams, because in reliability this is (may be) worse than a single-disk pool and bitrot on it - a single-bit error in a snapshot file can render it and all newer snapshots invalid and un-importable. Still, given enough scratch space on the sending and receiving sides and a bad (slow, glitchy) network in-between, I did go with compressed files of zfs-send streams (perhaps making recursion myself and using smaller files of one snapshot each - YMMV). For compression on multiCPU senders I can strongly suggest pigz --fast $filename (I did have problems in pigz-1.7.1 compressing several files with one command, maybe that's fixed now). If you're tight on space/transfer size more than on CPU, you can try other parallel algos - pbzip2, p7zip, etc. Likewise, you can also pass the file into an encryptor of your choice. Then I can rsync these files to the receiving server, using rsync -c and/or md5sum, sha256sum, sha1sum or whatever tool(s) of your liking to validate that the files received match those sent - better safe than sorry. I'm usually using rsync -cavPHK for any work, which gives you retryable transfers in case network goes down, status bar, directory recursion and hardlink support among other things. NFS is also retryable if so configured (even if the receiver gets rebooted in the process), and if you, for example, already have VPN between two sites, you might find it faster than rsync which involves extra encryption - maybe redundant in VPN case. When the scratch area on the receiver has got and validated the compressed snapshot stream, I can gzcat it and pipe into zfs recv. This ultimately validates that the zfs-send stream arrived intact and is fully receivable, and only then I can delete the temporary files involved - or retry the send from different steps (it is possible that the initial file was corrupted in RAM, etc.) Note that such approach via files essentially disables zfs-send deduplication which may be available in protocol between two active zfs commands, but AFAIK this does not preclude you from receiving data into deduped datasets - local dedup happens upon block writes anyway, like compression, encryption and stuff like that. HTH, //Jim Klimov __**_ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/**mailman/listinfo/zfs-discusshttp://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- Tiernan O'Toole blog.lotas-smartman.net www.geekphotographer.com www.tiernanotoole.ie ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
2012-10-05 13:13, Tiernan OToole wrote: Thanks for that Jim! Sounds like a plan there... One question about the storing ZFS dumps in a file... So, the idea of storing the data in a SFTP server which has an unknown underlying file system... Is that defiantly off limits, or can it be done? Mileages do vary. Maybe you should pack the stream files into archives with error-correction codes (or at least verification CRCs) like ZIP, RAR, likely p7zip, maybe others; and also keep checksum files. At least this can help detect or even fix small nasty surprises. The general concern is that zfs send streams have no built-in redundancy, I'm not sure about error-checking - likely it is there. And it is widely assumed that this being a stream, a small error can redirect the flow widely differently from expectations and cause the whole dataset state to be invalid (likely this snapshot receiving will be aborted, and then you can't receive any newer ones over it). That said, some people do keep the streams on tape; the NDMP tools and protocol from Sun IIRC do the same for backups. So it's not off-limits, but precautions may be due (keep 2+ copies, do CRC/ECC and so on). and should i be doing a full dump or just an incremental one? maybe incremental daily, and then a full dump weekly? A full dump of a large filesystem can be unbearably large for storage and transfers. Still, the idea of storing occasional full snapshots and a full history of incrementals (so that you can try to recover starting from any of the full snapshots you have) sounds sane. This way you have a sort of second copy by virtue of a full snapshot incorporating some state of the dataset and if the most recent one is broken - you can try to recover with the one(s) before it and applying more incremental snapshots. Likewise, errors in very old snapshots become irrelevant when a newer full snapshot is intact. But sometimes two or more things can break - or be detected to break - at once ;) In particular, regular drills should be done (and provisioned for) to test that you can in fact recover from your backups, and that they do contain all the data you need. Older configs can become obsolete as live systems evolve... HTH, //Jim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
Thanks again Jim. Very handy info. This is now my weekend project, so hopefully things go well! --Tiernan On Fri, Oct 5, 2012 at 10:40 AM, Jim Klimov jimkli...@cos.ru wrote: 2012-10-05 13:13, Tiernan OToole wrote: Thanks for that Jim! Sounds like a plan there... One question about the storing ZFS dumps in a file... So, the idea of storing the data in a SFTP server which has an unknown underlying file system... Is that defiantly off limits, or can it be done? Mileages do vary. Maybe you should pack the stream files into archives with error-correction codes (or at least verification CRCs) like ZIP, RAR, likely p7zip, maybe others; and also keep checksum files. At least this can help detect or even fix small nasty surprises. The general concern is that zfs send streams have no built-in redundancy, I'm not sure about error-checking - likely it is there. And it is widely assumed that this being a stream, a small error can redirect the flow widely differently from expectations and cause the whole dataset state to be invalid (likely this snapshot receiving will be aborted, and then you can't receive any newer ones over it). That said, some people do keep the streams on tape; the NDMP tools and protocol from Sun IIRC do the same for backups. So it's not off-limits, but precautions may be due (keep 2+ copies, do CRC/ECC and so on). and should i be doing a full dump or just an incremental one? maybe incremental daily, and then a full dump weekly? A full dump of a large filesystem can be unbearably large for storage and transfers. Still, the idea of storing occasional full snapshots and a full history of incrementals (so that you can try to recover starting from any of the full snapshots you have) sounds sane. This way you have a sort of second copy by virtue of a full snapshot incorporating some state of the dataset and if the most recent one is broken - you can try to recover with the one(s) before it and applying more incremental snapshots. Likewise, errors in very old snapshots become irrelevant when a newer full snapshot is intact. But sometimes two or more things can break - or be detected to break - at once ;) In particular, regular drills should be done (and provisioned for) to test that you can in fact recover from your backups, and that they do contain all the data you need. Older configs can become obsolete as live systems evolve... HTH, //Jim __**_ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/**mailman/listinfo/zfs-discusshttp://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- Tiernan O'Toole blog.lotas-smartman.net www.geekphotographer.com www.tiernanotoole.ie ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
On 10/05/12 21:36, Jim Klimov wrote: 2012-10-05 11:17, Tiernan OToole wrote: Also, as a follow up question, but slightly unrelated, when it comes to the ZFS Send, i could use SSH to do the send, directly to the machine... Or i could upload the compressed, and possibly encrypted dump to the server... Which, for resume-ability and speed, would be suggested? And if i where to go with an upload option, any suggestions on what i should use? As for this, the answer depends on network bandwidth, reliability, and snapshot file size - ultimately, on the probability and retry cost of an error during transmission. Many posters on the list strongly object to using files as storage for snapshot streams, because in reliability this is (may be) worse than a single-disk pool and bitrot on it - a single-bit error in a snapshot file can render it and all newer snapshots invalid and un-importable. Still, given enough scratch space on the sending and receiving sides and a bad (slow, glitchy) network in-between, I did go with compressed files of zfs-send streams (perhaps making recursion myself and using smaller files of one snapshot each - YMMV). For compression on multiCPU senders I can strongly suggest pigz --fast $filename (I did have problems in pigz-1.7.1 compressing several files with one command, maybe that's fixed now). If you're tight on space/transfer size more than on CPU, you can try other parallel algos - pbzip2, p7zip, etc. Likewise, you can also pass the file into an encryptor of your choice. I do have to suffer a slow, glitchy WAN to a remote server and rather than send stream files, I broke the data on the remote server into a more fine grained set of filesystems than I would do normally. In this case, I made the directories under what would have been the leaf filesystems filesystems themselves. By spreading the data over more filesystems, the individual incremental sends are smaller, so there is less data to resend if the link burps during a transfer. -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
Thanks Ian. That sounds like an option also. The plan was to break up the file systems anyway, since some i will want to be replicated remotely, and others not as much. --Tiernan On Fri, Oct 5, 2012 at 11:17 AM, Ian Collins i...@ianshome.com wrote: On 10/05/12 21:36, Jim Klimov wrote: 2012-10-05 11:17, Tiernan OToole wrote: Also, as a follow up question, but slightly unrelated, when it comes to the ZFS Send, i could use SSH to do the send, directly to the machine... Or i could upload the compressed, and possibly encrypted dump to the server... Which, for resume-ability and speed, would be suggested? And if i where to go with an upload option, any suggestions on what i should use? As for this, the answer depends on network bandwidth, reliability, and snapshot file size - ultimately, on the probability and retry cost of an error during transmission. Many posters on the list strongly object to using files as storage for snapshot streams, because in reliability this is (may be) worse than a single-disk pool and bitrot on it - a single-bit error in a snapshot file can render it and all newer snapshots invalid and un-importable. Still, given enough scratch space on the sending and receiving sides and a bad (slow, glitchy) network in-between, I did go with compressed files of zfs-send streams (perhaps making recursion myself and using smaller files of one snapshot each - YMMV). For compression on multiCPU senders I can strongly suggest pigz --fast $filename (I did have problems in pigz-1.7.1 compressing several files with one command, maybe that's fixed now). If you're tight on space/transfer size more than on CPU, you can try other parallel algos - pbzip2, p7zip, etc. Likewise, you can also pass the file into an encryptor of your choice. I do have to suffer a slow, glitchy WAN to a remote server and rather than send stream files, I broke the data on the remote server into a more fine grained set of filesystems than I would do normally. In this case, I made the directories under what would have been the leaf filesystems filesystems themselves. By spreading the data over more filesystems, the individual incremental sends are smaller, so there is less data to resend if the link burps during a transfer. -- Ian. __**_ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/**mailman/listinfo/zfs-discusshttp://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- Tiernan O'Toole blog.lotas-smartman.net www.geekphotographer.com www.tiernanotoole.ie ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Tiernan OToole I am in the process of planning a system which will have 2 ZFS servers, one on site, one off site. The on site server will be used by workstations and servers in house, and most of that will stay in house. There will, however, be data i want backed up somewhere else, which is where the offsite server comes in... This server will be sitting in a Data Center and will have some storage available to it (the whole server currently has 2 3Tb drives, though they are not dedicated to the ZFS box, they are on VMware ESXi). There is then some storage (currently 100Gb, but more can be requested) of SFTP enabled backup which i plan to use for some snapshots, but more on that later. Anyway, i want to confirm my plan and make sure i am not missing anything here... * build server in house with storage, pools, etc... * have a server in data center with enough storage for its reason, plus the extra for offsite backup * have one pool set as my offsite pool... anything in here should be backed up off site also... * possibly have another set as very offsite which will also be pushed to the SFTP server, but not sure... * give these pools out via SMB/NFS/iSCSI * every 6 or so hours take a snapshot of the 2 offsite pools. * do a ZFS send to the data center box * nightly, on the very offsite pool, do a ZFS send to the SFTP server * if anything goes wrong (my server dies, DC server dies, etc), Panic, download, pray... the usual... :) Anyway, I want to make sure i am doing this correctly... Is there anything on that list that sounds stupid or am i doing anything wrong? am i missing anything? Also, as a follow up question, but slightly unrelated, when it comes to the ZFS Send, i could use SSH to do the send, directly to the machine... Or i could upload the compressed, and possibly encrypted dump to the server... Which, for resume-ability and speed, would be suggested? And if i where to go with an upload option, any suggestions on what i should use? It is recommended, whenever possible, you should pipe the zfs send directly into a zfs receive on the receiving system. For two solid reasons: If a single bit is corrupted, the whole stream checksum is wrong and therefore the whole stream is rejected. So if this occurs, you want to detect it (in the form of one incremental failed) and then correct it (in the form of the next incremental succeeding). Whereas, if you store your streams on storage, it will go undetected, and everything after that point will be broken. If you need to do a restore, from a stream stored on storage, then your only choice is to restore the whole stream. You cannot look inside and just get one file. But if you had been doing send | receive, then you obviously can look inside the receiving filesystem and extract some individual specifics. If the recipient system doesn't support zfs receive, you might consider exporting an iscsi device, and allowing the sender system deal with it directly. Or share a filesystem (such as NFS) with the sender system, and let the sender create a recipient filesystem inside a file container, so the sender can deal with it directly. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
On Fri, Oct 5, 2012 at 3:17 AM, Ian Collins i...@ianshome.com wrote: I do have to suffer a slow, glitchy WAN to a remote server and rather than send stream files, I broke the data *on the remote server* into a more fine grained set of filesystems than I would do normally. In this case, I made the directories under what would have been the leaf filesystems filesystems themselves. Meaning you also broke the data on the LOCAL server into the same set of more granular filesystems? Or is it now possible to zfs send a subdirectory of a filesystem? ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Frank Cusack On Fri, Oct 5, 2012 at 3:17 AM, Ian Collins i...@ianshome.com wrote: I do have to suffer a slow, glitchy WAN to a remote server and rather than send stream files, I broke the data on the remote server into a more fine grained set of filesystems than I would do normally. In this case, I made the directories under what would have been the leaf filesystems filesystems themselves. Meaning you also broke the data on the LOCAL server into the same set of more granular filesystems? Or is it now possible to zfs send a subdirectory of a filesystem? zfs create instead of mkdir As Ian said - he didn't zfs send subdirs, he made filesystems where he otherwise would have used subdirs. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
On 10/06/12 07:57, Edward Ned Harvey (opensolarisisdeadlongliveopensolaris) wrote: From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Frank Cusack On Fri, Oct 5, 2012 at 3:17 AM, Ian Collinsi...@ianshome.com wrote: I do have to suffer a slow, glitchy WAN to a remote server and rather than send stream files, I broke the data on the remote server into a more fine grained set of filesystems than I would do normally. In this case, I made the directories under what would have been the leaf filesystems filesystems themselves. Meaning you also broke the data on the LOCAL server into the same set of more granular filesystems? Or is it now possible to zfs send a subdirectory of a filesystem? zfs create instead of mkdir As Ian said - he didn't zfs send subdirs, he made filesystems where he otherwise would have used subdirs. That right. I do have a lot of what would appear to be unnecessary filesystems, but after loosing the WAN 3 days into a large transfer, a change of tactic was required! -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Building an On-Site and Off-Size ZFS server, replication question
On Fri, Oct 5, 2012 at 1:28 PM, Ian Collins i...@ianshome.com wrote: I do have a lot of what would appear to be unnecessary filesystems, but after loosing the WAN 3 days into a large transfer, a change of tactic was required! I've recently (last year or so) gone the other way, and have made an effort to combine filesystems. I'm now thinking of remote replication so maybe I'll break them up again. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
