Re: [pfSense] ZFS on 2.4.2
On Thu, Mar 8, 2018 at 3:00 PM, Walter Parker wrote: > Are the FreeBSD 10.2 instructions ( > https://www.netgate.com/docs/platforms/rcc-dff-2220/freebsd.html) still > valid for 11.1? > > >- Connect the console cable (I have that setup) >- Boot from from a memstick image plugged into the USB port >- From the Menu select 3, Escape to the loader prompt >- Enter the following commands > - set comconsole_port=0x2F8 > - set comconsole_speed=38400 > - set hint.uart.0.flags=0x0 > - set hint.uart.1.flags=0x10 > - set console=comconsole > - boot >- Select shell or LiveCD from the FreeBSD installer menu >- Run tunefs > > Or does the 2.4 memstick installer give one an escape to shell option? > The hint lines for uart flags are unnecessary but harmless since FreeBSD 10. The image does have a "live" mode where it runs entirely in ramdisk, but nothing will let you set the serial port to the second port. You will have to use these settings to use the second port. You could try just booting to single user mode and run the tunefs. I don't remember if that works or not for the boot volume with FreeBSD 11. ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] ZFS on 2.4.2
On Thu, Mar 8, 2018 at 10:12 AM, Vick Khera wrote: > On Thu, Mar 8, 2018 at 11:10 AM, Zandr Milewski > wrote: > > > As someone who has spent easily 100 hours troubleshooting, rebuilding, > and > > restoring UFS based Netgate boxes that have to function in environments > > with less-that-datacenter grade power availability, I'll take "potential > > corruption in corner cases" over "1 in 4 chance it won't come back from a > > power cycle" > > > > *Any* journaled filesystem is an improvement. > > > > Journaling on UFS is just one setting away. Boot single user from USB, then > run "tunefs -j enable /dev/da0" for your boot device da0. Done. I don't > know why FreeBSD does not recommend this for the boot volume, but I think > as long as you never fill up the disk you're ok. I've no had issues with > it. > __ > That is an interesting idea. As I bought mine directly from the hardware store, I don't install pfSense myself. I've never booted it from USB. As this system doesn't have VGA, I may not be able to use a standard FreeBSD image out of the box. Are the FreeBSD 10.2 instructions ( https://www.netgate.com/docs/platforms/rcc-dff-2220/freebsd.html) still valid for 11.1? - Connect the console cable (I have that setup) - Boot from from a memstick image plugged into the USB port - From the Menu select 3, Escape to the loader prompt - Enter the following commands - set comconsole_port=0x2F8 - set comconsole_speed=38400 - set hint.uart.0.flags=0x0 - set hint.uart.1.flags=0x10 - set console=comconsole - boot - Select shell or LiveCD from the FreeBSD installer menu - Run tunefs Or does the 2.4 memstick installer give one an escape to shell option? Walter > _ > pfSense mailing list > https://lists.pfsense.org/mailman/listinfo/list > Support the project with Gold! https://pfsense.org/gold > -- The greatest dangers to liberty lurk in insidious encroachment by men of zeal, well-meaning but without understanding. -- Justice Louis D. Brandeis ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] ZFS on 2.4.2
On Thu, Mar 8, 2018 at 11:10 AM, Zandr Milewski wrote: > As someone who has spent easily 100 hours troubleshooting, rebuilding, and > restoring UFS based Netgate boxes that have to function in environments > with less-that-datacenter grade power availability, I'll take "potential > corruption in corner cases" over "1 in 4 chance it won't come back from a > power cycle" > > *Any* journaled filesystem is an improvement. > Journaling on UFS is just one setting away. Boot single user from USB, then run "tunefs -j enable /dev/da0" for your boot device da0. Done. I don't know why FreeBSD does not recommend this for the boot volume, but I think as long as you never fill up the disk you're ok. I've no had issues with it. ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] ZFS on 2.4.2
As someone who has spent easily 100 hours troubleshooting, rebuilding, and restoring UFS based Netgate boxes that have to function in environments with less-that-datacenter grade power availability, I'll take "potential corruption in corner cases" over "1 in 4 chance it won't come back from a power cycle" *Any* journaled filesystem is an improvement. On 3/8/18 06:19, Vick Khera wrote: On Wed, Mar 7, 2018 at 8:18 PM, Walter Parker wrote: don't use ECC. Can anyone show why my solution should switch file systems (given that I'm keeping my existing hardware) without changing the subject? I've read many of the scare stories from FreeNAS and they all seem to end up as a call to authority or a "fine, risk your data" without actually answering the question. The most important feature I use in ZFS is the snapshots. Combined cleverly with datasets and quotas, they make for very easy management of disk resources when needed. The FreeNAS model of boot environments is awesome, and I hope pfSense takes those up as well. It makes upgrades less stressful when you can just click a button to revert. As for the ECC, see this study https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35162.pdf for example. It is slightly old, but RAM hardware is not that much advanced since then. Basically, if you have a few gigs of RAM in your machine, it *will* produce bit errors. There are other studies that back this up too, and they are more recent. Personally, I don't understand why any computer, desktop or server, made these days is without ECC. My desktop has 16GB RAM with room for 16 more. I'm sure there are flipped bits in some of my work somewhere, but I'll never really know. If I'm lucky, the flipped bits are on unused sections of code loaded from the disk into RAM. ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] ZFS on 2.4.2
On Wed, Mar 7, 2018 at 8:18 PM, Walter Parker wrote: > don't use ECC. Can anyone show why my solution should switch file systems > (given that I'm keeping my existing hardware) without changing the subject? > I've read many of the scare stories from FreeNAS and they all seem to end > up as a call to authority or a "fine, risk your data" without actually > answering the question. > > The most important feature I use in ZFS is the snapshots. Combined cleverly with datasets and quotas, they make for very easy management of disk resources when needed. The FreeNAS model of boot environments is awesome, and I hope pfSense takes those up as well. It makes upgrades less stressful when you can just click a button to revert. As for the ECC, see this study https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35162.pdf for example. It is slightly old, but RAM hardware is not that much advanced since then. Basically, if you have a few gigs of RAM in your machine, it *will* produce bit errors. There are other studies that back this up too, and they are more recent. Personally, I don't understand why any computer, desktop or server, made these days is without ECC. My desktop has 16GB RAM with room for 16 more. I'm sure there are flipped bits in some of my work somewhere, but I'll never really know. If I'm lucky, the flipped bits are on unused sections of code loaded from the disk into RAM. ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] ZFS on 2.4.2
On Wed, Mar 7, 2018 at 2:31 PM, Peder Rovelstad wrote: > > That is an urban legend. One of original developers of ZFS was > > interviewed > > OK, then. Not my data. Best of luck. > > I've had other ZFS servers without ECC that have run successfully for several years. I know the risks and issues. While, yes, servers should run with ECC, the idea that that that ZFS requires ECC appears to a scare story to get people to buy ECC hardware. From my research over the last 10 years, I would say that 98% of the people sharing this information are passing on a scary story that someone else told them. This is a like the urban legends that we used to tell around the camp fire. Note, urban legends still get told and believed. You heard the one about flashing headlights, some people still tell and believe that story today. The closest I've seen to a reason for why it matters to ZFS is that it is one of the few file systems that can actually tell you when your data is corrupt before as well as after the fact. It solves many data issues and people seem to have taken that and require that the rest of the system be as robust as ZFS. When asked to present actual/real data as to why someone should use UFS instead of ZFS on a non-ECC system, I notice that the conversation changes from file systems to don't store data on systems that don't use ECC. Can anyone show why my solution should switch file systems (given that I'm keeping my existing hardware) without changing the subject? I've read many of the scare stories from FreeNAS and they all seem to end up as a call to authority or a "fine, risk your data" without actually answering the question. Does any make a standalone pfSense compatible router that is low power and not expensive [<$300] with enough ECC [or any ECC] memory? What would you do on a home budget to get multiple local backups of a multi-TB file server if you didn't have deep corporate pockets? I have the Netgate router, it is a real nice box. I don't see why using ZFS on it in addition to the other systems I have should be an issue, but there seem to be lots of cooks in the kitchen giving advice without sampling the product or explaining how they know there is a problem. Walter > ___ > pfSense mailing list > https://lists.pfsense.org/mailman/listinfo/list > Support the project with Gold! https://pfsense.org/gold > -- The greatest dangers to liberty lurk in insidious encroachment by men of zeal, well-meaning but without understanding. -- Justice Louis D. Brandeis ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] ZFS on 2.4.2
> That is an urban legend. One of original developers of ZFS was > interviewed OK, then. Not my data. Best of luck. ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] ZFS on 2.4.2
On Wed, Mar 7, 2018 at 2:04 PM, Walter Parker wrote: > without ECC. If there is a time bomb, then it exists for all file systems > running on computers without ECC. As this one of multiple backups for the > system, the risks are acceptable. > > If you have an actual failure method that makes ZFS worse, I'd love to see > the details. Then I could publish a paper and be "Internet famous. Yes, this is true. However, other file systems do not offer *any* hint of telling you when your data is corrupt on the platter like ZFS will. So if you know you don't have ECC protection, then you should not expect your data to be protected end to end. If you have ECC and a "regular" file system, the same is true. You just never know. ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] ZFS on 2.4.2
On Wed, Mar 7, 2018 at 7:36 AM, Peder Rovelstad wrote: > OH, and w/o ECC memory, it's a time bomb. > > That is an urban legend. One of original developers of ZFS was interviewed and asked about the "Scrub of Death", he said that ZFS doesn't fail in that way. ZFS is no worse than any other file system when running on a system without ECC. If there is a time bomb, then it exists for all file systems running on computers without ECC. As this one of multiple backups for the system, the risks are acceptable. If you have an actual failure method that makes ZFS worse, I'd love to see the details. Then I could publish a paper and be "Internet famous." Walter > -Original Message- > From: List [mailto:list-boun...@lists.pfsense.org] On Behalf Of Peder > Rovelstad > Sent: Wednesday, March 7, 2018 9:33 AM > To: 'pfSense Support and Discussion Mailing List' > Subject: Re: [pfSense] ZFS on 2.4.2 > > Oh, for certain. Lz4 compression is certainly stressful enough (too much > actually) for as low power a device as a SG-2220. > > Only posting to fan the flames! :) > > -Original Message- > From: List [mailto:list-boun...@lists.pfsense.org] On Behalf Of Vick Khera > Sent: Wednesday, March 7, 2018 8:57 AM > To: pfSense Support and Discussion Mailing List > Subject: Re: [pfSense] ZFS on 2.4.2 > > On Tue, Mar 6, 2018 at 6:51 PM, Peder Rovelstad > wrote: > > > Here's a ZFS tuning guide if you have not seen. > > https://wiki.freebsd.org/ZFSTuningGuide > > > > But only goes to v9. > > > > You 100% do not want nor need to turn on de-dupe. Especially on a boot > volume of pfSense. > ___ > pfSense mailing list > https://lists.pfsense.org/mailman/listinfo/list > Support the project with Gold! https://pfsense.org/gold > > ___ > pfSense mailing list > https://lists.pfsense.org/mailman/listinfo/list > Support the project with Gold! https://pfsense.org/gold > > ___ > pfSense mailing list > https://lists.pfsense.org/mailman/listinfo/list > Support the project with Gold! https://pfsense.org/gold > -- The greatest dangers to liberty lurk in insidious encroachment by men of zeal, well-meaning but without understanding. -- Justice Louis D. Brandeis ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] ZFS on 2.4.2
OH, and w/o ECC memory, it's a time bomb. -Original Message- From: List [mailto:list-boun...@lists.pfsense.org] On Behalf Of Peder Rovelstad Sent: Wednesday, March 7, 2018 9:33 AM To: 'pfSense Support and Discussion Mailing List' Subject: Re: [pfSense] ZFS on 2.4.2 Oh, for certain. Lz4 compression is certainly stressful enough (too much actually) for as low power a device as a SG-2220. Only posting to fan the flames! :) -Original Message- From: List [mailto:list-boun...@lists.pfsense.org] On Behalf Of Vick Khera Sent: Wednesday, March 7, 2018 8:57 AM To: pfSense Support and Discussion Mailing List Subject: Re: [pfSense] ZFS on 2.4.2 On Tue, Mar 6, 2018 at 6:51 PM, Peder Rovelstad wrote: > Here's a ZFS tuning guide if you have not seen. > https://wiki.freebsd.org/ZFSTuningGuide > > But only goes to v9. > You 100% do not want nor need to turn on de-dupe. Especially on a boot volume of pfSense. ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] ZFS on 2.4.2
Oh, for certain. Lz4 compression is certainly stressful enough (too much actually) for as low power a device as a SG-2220. Only posting to fan the flames! :) -Original Message- From: List [mailto:list-boun...@lists.pfsense.org] On Behalf Of Vick Khera Sent: Wednesday, March 7, 2018 8:57 AM To: pfSense Support and Discussion Mailing List Subject: Re: [pfSense] ZFS on 2.4.2 On Tue, Mar 6, 2018 at 6:51 PM, Peder Rovelstad wrote: > Here's a ZFS tuning guide if you have not seen. > https://wiki.freebsd.org/ZFSTuningGuide > > But only goes to v9. > You 100% do not want nor need to turn on de-dupe. Especially on a boot volume of pfSense. ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] ZFS on 2.4.2
On Tue, Mar 6, 2018 at 6:51 PM, Peder Rovelstad wrote: > Here's a ZFS tuning guide if you have not seen. > https://wiki.freebsd.org/ZFSTuningGuide > > But only goes to v9. > You 100% do not want nor need to turn on de-dupe. Especially on a boot volume of pfSense. ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] ZFS on 2.4.2
Here's a ZFS tuning guide if you have not seen. https://wiki.freebsd.org/ZFSTuningGuide But only goes to v9. Down the page they ref 2-5GB/TB for dedupe. Free advice, worth every penny paid! https://www.freebsd.org/doc/en/books/faq/all-about-zfs.html My NAS4Free server uses 90% of its 4GB RAM for a 3TB volume, configured with 1.75GB arc_max. -Original Message- From: List [mailto:list-boun...@lists.pfsense.org] On Behalf Of Paul Mather Sent: Tuesday, March 6, 2018 12:09 PM To: pfSense Support and Discussion Mailing List Subject: Re: [pfSense] ZFS on 2.4.2 On Mar 6, 2018, at 12:39 PM, Walter Parker < <mailto:walt...@gmail.com> walt...@gmail.com> wrote: > On Mon, Mar 5, 2018 at 6:38 PM, Curtis Maurand < <mailto:cmaur...@xyonet.com> cmaur...@xyonet.com> wrote: > >> ZFS is a memory hog. you need 1 GB of RAM for each TB of disk. > > > Curtis, can you provide some more details? I have been testing this > for the last couple of weeks and ZFS doesn't require 1G for each TB to > function (which is the standard meaning of need). > From my direct testing and experience 1G per TB is a rule of thumb for > suggested memory sizing on general purpose servers. Do you have > specific information that violating this rule of thumb will cause functional issues? > > To be more blunt, was this a case of drive by nerd sniping or do you > know something that will cause my specific use case to fail at some > point in the future? The "1G for each TB" sounds like the rule of thumb for when you plan to enable deduplication on a dataset. ZFS deduplication can be a disastrous memory hog (or else completely ruin your performance if you don't have sufficient ARC memory/resources), which is why many people do not enable it unless they've made a serious conscious decision to do so. I ran ZFS on a 1--2 GB RAM FreeBSD/i386 system for years and it was stable. I have to tune KVM and restrict ARC RAM consumption, but once I did that I had no problems. It's my experience that ZFS is more stable and tested on FreeBSD/amd64. Cheers, Paul. > > > Walter > > > >> On 3/1/2018 1:49 AM, Walter Parker wrote: >> >>> Forgot to CC the list. >>> >>> On Wed, Feb 28, 2018 at 10:13 PM, Walter Parker < <mailto:walt...@gmail.com> walt...@gmail.com> >>> wrote: >>> >>> Thank you for the backup script. >>>> >>>> By my calculations, 2G should be enough. If I limit the ARC cache >>>> to 1G, that leaves 1G for applications & kernel memory. As I'm not >>>> serving the 6TB drive up as a file server, but using it for one >>>> specific task (to receive the backups from one host) I figure that >>>> I don't need lots of memory. ZFS as a quick file server or busy >>>> server needs lots of memory to be quick. >>>> I've seen testing showing ZFS doing fast file copies on as little >>>> as 768M total system after proper memory tuning. >>>> >>>> I need ZFS because it is the only file system that can receive >>>> incremental ZFS snapshots and apply them. I have not setup the ZFS >>>> backup software yes, so I'm just using rsnapshot. First time it >>>> ran, it filled all 1G of the cache. I rebooted the firewall >>>> afterwards and now ZFS with 60-100M of usage (the amount of data >>>> that rsync updates on a daily basis is pretty small). >>>> Right now, the data from the other server is ~8.8G, compressed to >>>> 1.7G with lz4. >>>> >>>> When I get the full backup running, I will be ~1.5TB in size. ZFS >>>> snapshots should be pretty small and quick (as it can send just the >>>> data that was updated without having to walk the entire >>>> filesystem). An rsync backup would have to walk the whole system to find all of the changes. >>>> Most >>>> of the data on the system doesn't change (as it is a media library). >>>> >>>> I'll post back more results if people are interested, after I get >>>> the backup software working (I'm thinking about using ZapZend). >>>> >>>> >>>> Walter >>>> >>>> >>>> >>>> On Wed, Feb 28, 2018 at 8:54 PM, ED Fochler >>>> < <mailto:soek...@liquidbinary.com> soek...@liquidbinary.com> >>>> wrote: >>>> >>>> I feel like I'm late in responding to this, but I have to say that &g
Re: [pfSense] ZFS on 2.4.2
On Mar 6, 2018, at 12:39 PM, Walter Parker wrote: > On Mon, Mar 5, 2018 at 6:38 PM, Curtis Maurand wrote: > >> ZFS is a memory hog. you need 1 GB of RAM for each TB of disk. > > > Curtis, can you provide some more details? I have been testing this for the > last couple of weeks and ZFS doesn't require 1G for each TB to function > (which is the standard meaning of need). > From my direct testing and experience 1G per TB is a rule of thumb for > suggested memory sizing on general purpose servers. Do you have specific > information that violating this rule of thumb will cause functional issues? > > To be more blunt, was this a case of drive by nerd sniping or do you know > something that will cause my specific use case to fail at some point in the > future? The "1G for each TB" sounds like the rule of thumb for when you plan to enable deduplication on a dataset. ZFS deduplication can be a disastrous memory hog (or else completely ruin your performance if you don't have sufficient ARC memory/resources), which is why many people do not enable it unless they've made a serious conscious decision to do so. I ran ZFS on a 1--2 GB RAM FreeBSD/i386 system for years and it was stable. I have to tune KVM and restrict ARC RAM consumption, but once I did that I had no problems. It's my experience that ZFS is more stable and tested on FreeBSD/amd64. Cheers, Paul. > > > Walter > > > >> On 3/1/2018 1:49 AM, Walter Parker wrote: >> >>> Forgot to CC the list. >>> >>> On Wed, Feb 28, 2018 at 10:13 PM, Walter Parker >>> wrote: >>> >>> Thank you for the backup script. By my calculations, 2G should be enough. If I limit the ARC cache to 1G, that leaves 1G for applications & kernel memory. As I'm not serving the 6TB drive up as a file server, but using it for one specific task (to receive the backups from one host) I figure that I don't need lots of memory. ZFS as a quick file server or busy server needs lots of memory to be quick. I've seen testing showing ZFS doing fast file copies on as little as 768M total system after proper memory tuning. I need ZFS because it is the only file system that can receive incremental ZFS snapshots and apply them. I have not setup the ZFS backup software yes, so I'm just using rsnapshot. First time it ran, it filled all 1G of the cache. I rebooted the firewall afterwards and now ZFS with 60-100M of usage (the amount of data that rsync updates on a daily basis is pretty small). Right now, the data from the other server is ~8.8G, compressed to 1.7G with lz4. When I get the full backup running, I will be ~1.5TB in size. ZFS snapshots should be pretty small and quick (as it can send just the data that was updated without having to walk the entire filesystem). An rsync backup would have to walk the whole system to find all of the changes. Most of the data on the system doesn't change (as it is a media library). I'll post back more results if people are interested, after I get the backup software working (I'm thinking about using ZapZend). Walter On Wed, Feb 28, 2018 at 8:54 PM, ED Fochler wrote: I feel like I'm late in responding to this, but I have to say that 2GB of > RAM doesn't seem like nearly enough for a 6TB zfs volume. ZFS is great > in > a lot of ways, but is a RAM consuming monster. For something RAM > limited > like the 2220 I'd use a different, simpler file format. Then I'd use > rsync > based snapshots. > > Here's my personal backup script. :-) I haven't tried it FROM pfsense, > but I've used it to back up pfsense. > > ED. > > > > > > On 2018, Feb 21, at 12:23 PM, Walter Parker wrote: >> >> Hi, >> >> I have 2.4.2 installed on an SG-2220 from Netgate [nice box]. I just >> > bought > >> a 6TB powered USB drive from Costco and it works great (the drive has >> > its > >> own power supply and a USB hub). I want to use it take ZFS backups from >> > my > >> home server. >> >> I edited /boot/loader.conf.local and /etc/rc.conf.local to load ZFS on >> > boot > >> and created a pool and a file system. That worked, but the memory ran >> > low > >> so I restricted the ARC cache to 1G to keep a bit more memory free and >> rebooted. When the system rebooted it did not remount the pool (and >> therefore the file system) because the pool what marked as in use by >> another system (itself). That means that the pool was not properly >> exported/umounted at shutdown. >> >> Taking a quick look a rc.shutdown, I notice that it calls a customized >> pfsense shutdown script at the beginning and then exits. Is there a >> good
Re: [pfSense] ZFS on 2.4.2
On Mon, Mar 5, 2018 at 6:38 PM, Curtis Maurand wrote: > ZFS is a memory hog. you need 1 GB of RAM for each TB of disk. Curtis, can you provide some more details? I have been testing this for the last couple of weeks and ZFS doesn't require 1G for each TB to function (which is the standard meaning of need). >From my direct testing and experience 1G per TB is a rule of thumb for suggested memory sizing on general purpose servers. Do you have specific information that violating this rule of thumb will cause functional issues? To be more blunt, was this a case of drive by nerd sniping or do you know something that will cause my specific use case to fail at some point in the future? Walter > On 3/1/2018 1:49 AM, Walter Parker wrote: > >> Forgot to CC the list. >> >> On Wed, Feb 28, 2018 at 10:13 PM, Walter Parker >> wrote: >> >> Thank you for the backup script. >>> >>> By my calculations, 2G should be enough. If I limit the ARC cache to 1G, >>> that leaves 1G for applications & kernel memory. As I'm not serving the >>> 6TB >>> drive up as a file server, but using it for one specific task (to receive >>> the backups from one host) I figure that I don't need lots of memory. ZFS >>> as a quick file server or busy server needs lots of memory to be quick. >>> I've seen testing showing ZFS doing fast file copies on as little as 768M >>> total system after proper memory tuning. >>> >>> I need ZFS because it is the only file system that can receive >>> incremental >>> ZFS snapshots and apply them. I have not setup the ZFS backup software >>> yes, >>> so I'm just using rsnapshot. First time it ran, it filled all 1G of the >>> cache. I rebooted the firewall afterwards and now ZFS with 60-100M of >>> usage >>> (the amount of data that rsync updates on a daily basis is pretty small). >>> Right now, the data from the other server is ~8.8G, compressed to 1.7G >>> with >>> lz4. >>> >>> When I get the full backup running, I will be ~1.5TB in size. ZFS >>> snapshots should be pretty small and quick (as it can send just the data >>> that was updated without having to walk the entire filesystem). An rsync >>> backup would have to walk the whole system to find all of the changes. >>> Most >>> of the data on the system doesn't change (as it is a media library). >>> >>> I'll post back more results if people are interested, after I get the >>> backup software working (I'm thinking about using ZapZend). >>> >>> >>> Walter >>> >>> >>> >>> On Wed, Feb 28, 2018 at 8:54 PM, ED Fochler >>> wrote: >>> >>> I feel like I'm late in responding to this, but I have to say that 2GB of RAM doesn't seem like nearly enough for a 6TB zfs volume. ZFS is great in a lot of ways, but is a RAM consuming monster. For something RAM limited like the 2220 I'd use a different, simpler file format. Then I'd use rsync based snapshots. Here's my personal backup script. :-) I haven't tried it FROM pfsense, but I've used it to back up pfsense. ED. On 2018, Feb 21, at 12:23 PM, Walter Parker wrote: > > Hi, > > I have 2.4.2 installed on an SG-2220 from Netgate [nice box]. I just > bought > a 6TB powered USB drive from Costco and it works great (the drive has > its > own power supply and a USB hub). I want to use it take ZFS backups from > my > home server. > > I edited /boot/loader.conf.local and /etc/rc.conf.local to load ZFS on > boot > and created a pool and a file system. That worked, but the memory ran > low > so I restricted the ARC cache to 1G to keep a bit more memory free and > rebooted. When the system rebooted it did not remount the pool (and > therefore the file system) because the pool what marked as in use by > another system (itself). That means that the pool was not properly > exported/umounted at shutdown. > > Taking a quick look a rc.shutdown, I notice that it calls a customized > pfsense shutdown script at the beginning and then exits. Is there a > good > place in the configuration where I can put/call the proper zfs shutdown > script so that the pool is properly stopped/exported so that it imports > correctly on boot? > > > Walter > > -- > The greatest dangers to liberty lurk in insidious encroachment by men > of > zeal, well-meaning but without understanding. -- Justice Louis D. > Brandeis > ___ > pfSense mailing list > https://lists.pfsense.org/mailman/listinfo/list > Support the project with Gold! https://pfsense.org/gold > >>> -- >>> The greatest dangers to liberty lurk in insidious encroachment by men of >>> zeal, well-meaning but without understanding. -- Justice Louis D. >>> Brandeis >>> >>> >> >> > -- > Best Regards > Curtis Maurand > Princip
Re: [pfSense] ZFS on 2.4.2
ZFS is a memory hog. you need 1 GB of RAM for each TB of disk. On 3/1/2018 1:49 AM, Walter Parker wrote: Forgot to CC the list. On Wed, Feb 28, 2018 at 10:13 PM, Walter Parker wrote: Thank you for the backup script. By my calculations, 2G should be enough. If I limit the ARC cache to 1G, that leaves 1G for applications & kernel memory. As I'm not serving the 6TB drive up as a file server, but using it for one specific task (to receive the backups from one host) I figure that I don't need lots of memory. ZFS as a quick file server or busy server needs lots of memory to be quick. I've seen testing showing ZFS doing fast file copies on as little as 768M total system after proper memory tuning. I need ZFS because it is the only file system that can receive incremental ZFS snapshots and apply them. I have not setup the ZFS backup software yes, so I'm just using rsnapshot. First time it ran, it filled all 1G of the cache. I rebooted the firewall afterwards and now ZFS with 60-100M of usage (the amount of data that rsync updates on a daily basis is pretty small). Right now, the data from the other server is ~8.8G, compressed to 1.7G with lz4. When I get the full backup running, I will be ~1.5TB in size. ZFS snapshots should be pretty small and quick (as it can send just the data that was updated without having to walk the entire filesystem). An rsync backup would have to walk the whole system to find all of the changes. Most of the data on the system doesn't change (as it is a media library). I'll post back more results if people are interested, after I get the backup software working (I'm thinking about using ZapZend). Walter On Wed, Feb 28, 2018 at 8:54 PM, ED Fochler wrote: I feel like I'm late in responding to this, but I have to say that 2GB of RAM doesn't seem like nearly enough for a 6TB zfs volume. ZFS is great in a lot of ways, but is a RAM consuming monster. For something RAM limited like the 2220 I'd use a different, simpler file format. Then I'd use rsync based snapshots. Here's my personal backup script. :-) I haven't tried it FROM pfsense, but I've used it to back up pfsense. ED. On 2018, Feb 21, at 12:23 PM, Walter Parker wrote: Hi, I have 2.4.2 installed on an SG-2220 from Netgate [nice box]. I just bought a 6TB powered USB drive from Costco and it works great (the drive has its own power supply and a USB hub). I want to use it take ZFS backups from my home server. I edited /boot/loader.conf.local and /etc/rc.conf.local to load ZFS on boot and created a pool and a file system. That worked, but the memory ran low so I restricted the ARC cache to 1G to keep a bit more memory free and rebooted. When the system rebooted it did not remount the pool (and therefore the file system) because the pool what marked as in use by another system (itself). That means that the pool was not properly exported/umounted at shutdown. Taking a quick look a rc.shutdown, I notice that it calls a customized pfsense shutdown script at the beginning and then exits. Is there a good place in the configuration where I can put/call the proper zfs shutdown script so that the pool is properly stopped/exported so that it imports correctly on boot? Walter -- The greatest dangers to liberty lurk in insidious encroachment by men of zeal, well-meaning but without understanding. -- Justice Louis D. Brandeis ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold -- The greatest dangers to liberty lurk in insidious encroachment by men of zeal, well-meaning but without understanding. -- Justice Louis D. Brandeis -- Best Regards Curtis Maurand Principal Xyonet Web Hosting mailto:cmaur...@xyonet.com http://www.xyonet.com ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] ZFS on 2.4.2
Here's my simple backup script function. Just stick it into a /bin/sh script (should work in bash too) and call it once per pfSense instance. I've been using this for years to backup my production firewalls. pfsense_config() { local FWNAME FWURL FWPASS CSRF CSRF2 COOKIEFILE PFDATE FWNAME="$1" FWPASS="$2" FWURL="https://${FWNAME}"; COOKIEFILE=`mktemp -t cookies` PFDATE=`date +%Y%m%d%H%M%S` printf "Downloading Firewall Config for $FWNAME\n" CSRF=`curl -k -L -c ${COOKIEFILE} ${FWURL}/diag_backup.php | grep "name='__csrf_magic'" | head -1 | sed 's/.*value="\(.*\)".*/\1/'` CSRF2=`curl -k -L -c ${COOKIEFILE} -b ${COOKIEFILE} -d "login=Login&usernamefld=admin&passwordfld=$FWPASS&__csrf_magic=${CSRF}" ${FWURL}/diag_backup.php | grep "name='__csrf_magic'" | head -1 | sed 's/.*value="\(.*\)".*/\1/'` curl -k -b ${COOKIEFILE} -d "Submit=download&donotbackuprrd=checked&__csrf_magic=${CSRF2}" -o config-$FWNAME-$PFDATE.xml ${FWURL}/diag_backup.php rm -f ${COOKIEFILE} } You call it like this: pfsense_config firewall.example.com mySecr3tPassword and it stores the backup XML in a file based on the date and firewall name. ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] ZFS on 2.4.2
Forgot to CC the list. On Wed, Feb 28, 2018 at 10:13 PM, Walter Parker wrote: > Thank you for the backup script. > > By my calculations, 2G should be enough. If I limit the ARC cache to 1G, > that leaves 1G for applications & kernel memory. As I'm not serving the 6TB > drive up as a file server, but using it for one specific task (to receive > the backups from one host) I figure that I don't need lots of memory. ZFS > as a quick file server or busy server needs lots of memory to be quick. > I've seen testing showing ZFS doing fast file copies on as little as 768M > total system after proper memory tuning. > > I need ZFS because it is the only file system that can receive incremental > ZFS snapshots and apply them. I have not setup the ZFS backup software yes, > so I'm just using rsnapshot. First time it ran, it filled all 1G of the > cache. I rebooted the firewall afterwards and now ZFS with 60-100M of usage > (the amount of data that rsync updates on a daily basis is pretty small). > Right now, the data from the other server is ~8.8G, compressed to 1.7G with > lz4. > > When I get the full backup running, I will be ~1.5TB in size. ZFS > snapshots should be pretty small and quick (as it can send just the data > that was updated without having to walk the entire filesystem). An rsync > backup would have to walk the whole system to find all of the changes. Most > of the data on the system doesn't change (as it is a media library). > > I'll post back more results if people are interested, after I get the > backup software working (I'm thinking about using ZapZend). > > > Walter > > > > On Wed, Feb 28, 2018 at 8:54 PM, ED Fochler > wrote: > >> I feel like I'm late in responding to this, but I have to say that 2GB of >> RAM doesn't seem like nearly enough for a 6TB zfs volume. ZFS is great in >> a lot of ways, but is a RAM consuming monster. For something RAM limited >> like the 2220 I'd use a different, simpler file format. Then I'd use rsync >> based snapshots. >> >> Here's my personal backup script. :-) I haven't tried it FROM pfsense, >> but I've used it to back up pfsense. >> >> ED. >> >> >> >> >> >> > On 2018, Feb 21, at 12:23 PM, Walter Parker wrote: >> > >> > Hi, >> > >> > I have 2.4.2 installed on an SG-2220 from Netgate [nice box]. I just >> bought >> > a 6TB powered USB drive from Costco and it works great (the drive has >> its >> > own power supply and a USB hub). I want to use it take ZFS backups from >> my >> > home server. >> > >> > I edited /boot/loader.conf.local and /etc/rc.conf.local to load ZFS on >> boot >> > and created a pool and a file system. That worked, but the memory ran >> low >> > so I restricted the ARC cache to 1G to keep a bit more memory free and >> > rebooted. When the system rebooted it did not remount the pool (and >> > therefore the file system) because the pool what marked as in use by >> > another system (itself). That means that the pool was not properly >> > exported/umounted at shutdown. >> > >> > Taking a quick look a rc.shutdown, I notice that it calls a customized >> > pfsense shutdown script at the beginning and then exits. Is there a good >> > place in the configuration where I can put/call the proper zfs shutdown >> > script so that the pool is properly stopped/exported so that it imports >> > correctly on boot? >> > >> > >> > Walter >> > >> > -- >> > The greatest dangers to liberty lurk in insidious encroachment by men of >> > zeal, well-meaning but without understanding. -- Justice Louis D. >> Brandeis >> > ___ >> > pfSense mailing list >> > https://lists.pfsense.org/mailman/listinfo/list >> > Support the project with Gold! https://pfsense.org/gold >> >> >> > > > -- > The greatest dangers to liberty lurk in insidious encroachment by men of > zeal, well-meaning but without understanding. -- Justice Louis D. Brandeis > -- The greatest dangers to liberty lurk in insidious encroachment by men of zeal, well-meaning but without understanding. -- Justice Louis D. Brandeis ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] ZFS on 2.4.2
You don't need to export the pool on shutdown. Even an unclean shutdown should survive automatically on the reboot. I can't think of a reason ZFS would fail like you describe. On Wed, Feb 21, 2018 at 12:23 PM, Walter Parker wrote: > Hi, > > I have 2.4.2 installed on an SG-2220 from Netgate [nice box]. I just bought > a 6TB powered USB drive from Costco and it works great (the drive has its > own power supply and a USB hub). I want to use it take ZFS backups from my > home server. > > I edited /boot/loader.conf.local and /etc/rc.conf.local to load ZFS on boot > and created a pool and a file system. That worked, but the memory ran low > so I restricted the ARC cache to 1G to keep a bit more memory free and > rebooted. When the system rebooted it did not remount the pool (and > therefore the file system) because the pool what marked as in use by > another system (itself). That means that the pool was not properly > exported/umounted at shutdown. > > Taking a quick look a rc.shutdown, I notice that it calls a customized > pfsense shutdown script at the beginning and then exits. Is there a good > place in the configuration where I can put/call the proper zfs shutdown > script so that the pool is properly stopped/exported so that it imports > correctly on boot? > > > Walter > > -- > The greatest dangers to liberty lurk in insidious encroachment by men of > zeal, well-meaning but without understanding. -- Justice Louis D. > Brandeis > ___ > pfSense mailing list > https://lists.pfsense.org/mailman/listinfo/list > Support the project with Gold! https://pfsense.org/gold > ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
[pfSense] ZFS on 2.4.2
Hi, I have 2.4.2 installed on an SG-2220 from Netgate [nice box]. I just bought a 6TB powered USB drive from Costco and it works great (the drive has its own power supply and a USB hub). I want to use it take ZFS backups from my home server. I edited /boot/loader.conf.local and /etc/rc.conf.local to load ZFS on boot and created a pool and a file system. That worked, but the memory ran low so I restricted the ARC cache to 1G to keep a bit more memory free and rebooted. When the system rebooted it did not remount the pool (and therefore the file system) because the pool what marked as in use by another system (itself). That means that the pool was not properly exported/umounted at shutdown. Taking a quick look a rc.shutdown, I notice that it calls a customized pfsense shutdown script at the beginning and then exits. Is there a good place in the configuration where I can put/call the proper zfs shutdown script so that the pool is properly stopped/exported so that it imports correctly on boot? Walter -- The greatest dangers to liberty lurk in insidious encroachment by men of zeal, well-meaning but without understanding. -- Justice Louis D. Brandeis ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold