We do understand. We just go through this with people over and over. Humans 
tend to ask without reading. ;) Yes we were all in 'that' boat at done time, so 
our warnings are worth considering. 

I highly recommend ZFS for what you want. However some time should be taken. 
Just like adding large storage into a big company like Amazon - test, test 
again, abuse, test, test.....

I personally have 96 TB storage in the mad science lab and with clients 
combined, probably close to triple that. And ZFS has saved some clients a lot 
of stress and money where other tech has left them crying. 

We want you to be successful with ZFS. 

Jason
Sent from my iPhone 5S

> On May 20, 2014, at 2:28 PM, James Hoyt <djnati...@gmail.com> wrote:
> 
> Hi Bjorn thanks for your reply and thanks for your help Jason in all
> this. I've actually been in the IT industry for 12 years, A+
> certified, and currently pursuing my CCNA and MCSA so a technical
> setup didn't intimidate me. (granted servers are a new beast to me) I
> came across Mac ZFS while researching RAID options. As I'm getting new
> music in wav/flac daily from a number of sources, a manual backup
> system really wouldn't work for me as it's too hard to keep up. I
> tried once with a blu-ray writer and it was a nightmare.. plus I'm
> regularly categorizing music with MediaMonkey (why I have a virtual
> machine because it's Windows only... oh why won't iTunes support FLAC
> natively!) so all the tracks are updated now and then. So I thought an
> offsite backup that's updated every few months along with a four-drive
> RAID setup with one drive for redundancy would be all that I would
> need.
> 
> MacZFS.org is well put together and the tutorials isn't intimidating
> at all. Run a few terminal commands? I can do that. The depth of ZFS
> wasn't really covered nor did it really state that more research is
> needed (like if I need 4k setup.. or whatever that is D:  ) So it's
> rather frustrating to just find out I moved all my data off my
> individual 2/1.5 TB drives to find out I did it wrong when I was
> careful, very careful, to follow the Getting Started guide and FAQ
> precisely.
> 
> Bleh D:
> 
> Jason, I apologize for coming out rough. I felt like I was being
> treated like a lazy moron, which I'm not. I've researched a variety of
> RAID solutions quite a bit and thought I was all setup for ZFS and
> just had to change some configurations to speed it up. I'm sure you
> weren't born with this knowledge and needed the help of others to
> guide you in the right direction. I was googling things like "slow
> zpool ZFS" and other similar terms but just couldn't find anything
> concrete (because my problem isn't concrete).
> 
> I tried searching if my drives are 4k with no luck. I saw an article
> back from 2010 stating hard drives were planning to all be 4k in
> 2011... this leads me to believe that they are 4k since I purchased
> them new last year. Crap D: Is there a for sure way I can see if they
> are 4k? Could this be my performance issue or is it just because my
> directories have large amounts of folders/files in them?
> 
> Again, sorry if I came out rude. This is all new technology to me and
> I'm doing my best to become familiar with it.
> 
> Thank you,
> 
> James
> 
>> On Tue, May 20, 2014 at 12:52 PM, Daniel Becker <razzf...@gmail.com> wrote:
>> James,
>> 
>> Perhaps the takeaway here is that MacZFS (and arguably ZFS in general) is
>> really not a great fit for the casual user. ZFS is very powerful once you
>> take the time to really get familiar with it, but it does require a fair
>> amount of research to get started, and it gives you lots of ways to shoot
>> yourself in the foot. And as you found out yourself, there are a fair number
>> of caveats and behavioral oddities when running ZFS on a Mac. If you want
>> something that "just works" without digging into the details and that gives
>> you behavior just as you would expect it from other file systems, it's
>> probably not for you (at least not for anything other than experimentation).
>> 
>> I know that the MacZFS page likes to give a somewhat different impression,
>> but in my opinion encouraging non-technical users to install it is really
>> doing a disservice both to said users and to the community as a whole.
>> 
>> Daniel
>> 
>> 
>>> On Tue, May 20, 2014 at 9:59 AM, James Hoyt <djnati...@gmail.com> wrote:
>>> 
>>> I did status as you can see from my original post.. I didn't know
>>> scrub and clean. I did my research only on MacZFS because I thought
>>> that's only where it mattered. I didn't trust info on other sites
>>> because I didn't think it was relevant to how Mac ZFS operated.
>>> 
>>> Please show me where I could have found the scrub command on
>>> maczfs.org because it is not there. I see nothing about clean either.
>>> 
>>> I'm openly stating I don't know it and it's not stated on the wiki or
>>> FAQ or getting started section on maczfs.org. There is no refusal
>>> going on.
>>> 
>>> On Tue, May 20, 2014 at 11:51 AM, Jason Belec
>>> <jasonbe...@belecmartin.com> wrote:
>>>> Sorry you feel that way. We have had a lot of people in your situation.
>>>> You seem to have skipped over the basics.
>>>> 
>>>> Zpool scub murr
>>>> 
>>>> Zpool status murr
>>>> 
>>>> 
>>>> This command is on every ZFS site. Your openly stating you don't know it
>>>> and refuse to look it up. I wish you the best.
>>>> 
>>>> 
>>>> Jason
>>>> Sent from my iPhone 5S
>>>> 
>>>>> On May 20, 2014, at 12:09 PM, James Hoyt <djnati...@gmail.com> wrote:
>>>>> 
>>>>> You have completely lost me at this point. You were rather
>>>>> condescending and not helpful. I was hoping for instructions on how to
>>>>> clean and scrub and saw none of that. At least point me to some proper
>>>>> links. I also don't know what a 4k drive is.
>>>>> 
>>>>> I carefully followed and read ALL the instructions and FAQ and Getting
>>>>> Started guide on maczfs.org. Please don't speak to me like I didn't do
>>>>> my research or follow the proper instructions.
>>>>> 
>>>>> - James
>>>>> 
>>>>>> On Tue, May 20, 2014 at 9:24 AM, Jason Belec
>>>>>> <jasonbe...@belecmartin.com> wrote:
>>>>>> OK, one thing, any indexing under that version of ZFS is going to kill
>>>>>> performance. Long standing issue.
>>>>>> 
>>>>>> No backups? Did you bump your noggin? With your current setup you have
>>>>>> improved your chances if your scrubbing regularly and if you only lose a
>>>>>> drive at anyone time. And adding backup will drastically increase your
>>>>>> chances.
>>>>>> 
>>>>>> Not understanding ZFS is a BIG reason to stop and re-evaluate your
>>>>>> priorities. It's amazing tech IF used properly.
>>>>>> 
>>>>>> For what it sounds like you want from ZFS you should use mirrors. You
>>>>>> can do 2 mirrors of 2 drives each stripped under ZFS. This will increase 
>>>>>> the
>>>>>> safety of your data. Even that should have a back up drive you move key
>>>>>> files or better yet 'snapshots' onto.
>>>>>> 
>>>>>> BUT you are going to have to understand ZFS to have any hope of not
>>>>>> drowning in a pool of tears at some point.
>>>>>> 
>>>>>> The new ZFS is under development but far more functional. Eliminating
>>>>>> many of the old version issues listed numerous times throughout the 
>>>>>> forum.
>>>>>> Either way you should ALWAYS understand the tech you rely on. Period.
>>>>>> 
>>>>>> Please start learning with the word 'scrub' then the word 'snapshot'
>>>>>> and how to swap a failed drive and do it all. Before committing your
>>>>>> valuable data. Drives fail. Repeat. Drives fail.  Data must be restored 
>>>>>> at
>>>>>> some point. ZFS is magical if you have planned ahead. I have recovered 
>>>>>> data
>>>>>> assumed totally lost, YMMV.
>>>>>> 
>>>>>> As for those drives are they 4k? If so you formatted your pool
>>>>>> incorrectly. I don't have any of those so I don't have notes. Should be a
>>>>>> simple Google search to find out. And the wiki has the instructions on 4k
>>>>>> drive setup.
>>>>>> 
>>>>>> Doing things right is what the wiki tries to help people with. The
>>>>>> forum allows you to search for other peoples heartbreak to help prevent 
>>>>>> your
>>>>>> own.  The wizards tracking this stuff have done a wonderful job.
>>>>>> 
>>>>>> Hope this gets you rolling. I'd still check your cables as well.
>>>>>> Normally I attach a drive, build a pool, test a lot, destroy pool. Add
>>>>>> another drive. Repeat. Better safe than sorry. Manufacturers are not safe
>>>>>> guarding your data.
>>>>>> 
>>>>>> Jason
>>>>>> Sent from my iPhone 5S
>>>>>> 
>>>>>>> On May 20, 2014, at 9:37 AM, James Hoyt <djnati...@gmail.com> wrote:
>>>>>>> 
>>>>>>> Thanks for the detailed reply.
>>>>>>> 
>>>>>>> The slow performance is only when I'm using the RAID array so I
>>>>>>> assume
>>>>>>> without it connected means I can't use it means there is no slow
>>>>>>> performance. I would love instructions on how to scrub/clean the
>>>>>>> pool.
>>>>>>> Does it do a data wipe?
>>>>>>> 
>>>>>>> I was trying to think of a good backup solution. I have over 3 TBs of
>>>>>>> music in FLAC (lots of which I've paid for) and was hoping RAIDZ
>>>>>>> would
>>>>>>> take away the need for backups. I was thinking of buying a 4 TB drive
>>>>>>> and moving all my data on that and storing the drive offsite or
>>>>>>> something (in case of burglary, fires, etc). Having a single drive
>>>>>>> fail safe seems secure enough for me so I don't think incremental
>>>>>>> backups are needed.
>>>>>>> 
>>>>>>> As for running the latest beta ZFS, I didn't because the FAQ warned
>>>>>>> me
>>>>>>> not to. What are the differences? Would I have to format and rebuild
>>>>>>> the array?
>>>>>>> 
>>>>>>> The drives I have are four 3 TB Hitachi HDS723030BLE640.
>>>>>>> 
>>>>>>> I started navigating around my computer again, and the slowdown seems
>>>>>>> to be when going into folders with over 1000 files (for anything more
>>>>>>> it will take 1-3 minutes to just list the files in the directory).
>>>>>>> Also when I'm saving images from Firefox (no virtual machine running)
>>>>>>> it takes awhile to navigate the folder structure and sometimes not
>>>>>>> all
>>>>>>> the folders show, but they do in the Finder. So I wonder if this is
>>>>>>> an
>>>>>>> issue with programs not getting along with ZFS but the finder being
>>>>>>> fine with it.
>>>>>>> 
>>>>>>> Other things to note, I did disable Spotlight on the drive to make
>>>>>>> sure that isn't running, but I do have QuickSilver. Originally, I had
>>>>>>> QuickSilver indexing the drive, but the computer was practically
>>>>>>> unusable when it did that so I disabled that.
>>>>>>> 
>>>>>>> I look forward to any advice you guys may have.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> 
>>>>>>> James
>>>>>>> 
>>>>>>>> On Tue, May 20, 2014 at 6:14 AM, Jason Belec
>>>>>>>> <jasonbe...@belecmartin.com> wrote:
>>>>>>>> OK, doesn't look like RAM, processor etc., are the issue.... Let's
>>>>>>>> work with that in mind for now.
>>>>>>>> 
>>>>>>>> When the pool and the associated drives are not connected, is the
>>>>>>>> computer back to your expectation of normal? If so, you have one or 
>>>>>>>> more bad
>>>>>>>> cables, one or more bad drives, or a bit of both, perhaps a bad or not 
>>>>>>>> quite
>>>>>>>> capable power supply (solves 90% of all issues I come across). Maybe 
>>>>>>>> even an
>>>>>>>> issue with the motherboard. Simplest thing, have you run a scrub on 
>>>>>>>> this
>>>>>>>> pool? Clean?
>>>>>>>> 
>>>>>>>> The type of drives you have is not an issue, the make and known
>>>>>>>> issues with said drives might be, but you didn't provide that info.
>>>>>>>> 
>>>>>>>> Using a raidcard and macJournaled terms, thrown out will not help
>>>>>>>> you, your either ZFS or not. That said, you will not get the same 
>>>>>>>> speed from
>>>>>>>> ZFS as from other raid setups, but you will get peace of mind on data
>>>>>>>> integrity. I do hope you are also backing up data from the pool as 
>>>>>>>> well or
>>>>>>>> eventually you will be in tears like so many others. A little forum
>>>>>>>> searching under old and new versions of mac zfs will be helpful.
>>>>>>>> 
>>>>>>>> Since your getting started, once this is resolved it might be better
>>>>>>>> to build/run this under the latest (yes its in development) Mac ZFS 
>>>>>>>> rather
>>>>>>>> than the old tired version. It is quite a bit different, modern and 
>>>>>>>> makes
>>>>>>>> many things a lot easier. (Insert legal disclaimer here) ;)
>>>>>>>> 
>>>>>>>> Interesting aside:
>>>>>>>> Dave mentioned an interesting point about wearing out SSDs, and I
>>>>>>>> must admit I've had two such occurrences but only with a hackintosh 
>>>>>>>> and only
>>>>>>>> with less than stellar drives. Seems that here around the mad science 
>>>>>>>> lab
>>>>>>>> Intel SSDs are the most reliable long term. I have two of their 
>>>>>>>> originals
>>>>>>>> still outlasting several other brands.
>>>>>>>> 
>>>>>>>> --
>>>>>>>> Jason Belec
>>>>>>>> Sent from my iPad
>>>>>>>> 
>>>>>>>>> On May 19, 2014, at 10:05 AM, James Hoyt <djnati...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> Thanks for all the replies guys =D
>>>>>>>>> 
>>>>>>>>> Sorry for lack of information. I'm running a Hackintosh with a 256
>>>>>>>>> GB
>>>>>>>>> SSD and I sometimes run Windows 8.1 in a virtual machine via VmWare
>>>>>>>>> Fusion. The virtual image file is also located on the SSD. The only
>>>>>>>>> files I have on my zpool are data files. I don't run an OS or VM
>>>>>>>>> image
>>>>>>>>> from it. I have 12 GBs of RAM and a four core i5 processor. On the
>>>>>>>>> VM,
>>>>>>>>> I dedicate 6 GBs of RAM and 2 cores to it. It should be noted that
>>>>>>>>> I
>>>>>>>>> experience the slow down even when vmware is off it's just the
>>>>>>>>> drives
>>>>>>>>> act the slowest when the VM is running.
>>>>>>>>> 
>>>>>>>>> As for how I created the zpool, I followed the Getting Started
>>>>>>>>> guide with
>>>>>>>>> 
>>>>>>>>> zpool create murr raidz disk3s2 disk1s2 disk2s2 disk4s2
>>>>>>>>> 
>>>>>>>>> Please help... I really hope I don't have to recreate it, but it's
>>>>>>>>> looking that way.
>>>>>>>>> 
>>>>>>>>> Would it be better if I bought a RAID card and use Mac OS
>>>>>>>>> Journaled?
>>>>>>>>> Cost is an issue... the other issue is these are regular desktop
>>>>>>>>> 7200
>>>>>>>>> RPM drives.. not NAS drives.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> 
>>>>>>>>> James
>>>>>>>>> 
>>>>>>>>>> On Mon, May 19, 2014 at 7:43 AM, Jason Belec
>>>>>>>>>> <jasonbe...@belecmartin.com> wrote:
>>>>>>>>>> Dave has posted some good info. Reminds me why I prefer
>>>>>>>>>> Virtualbox. ;) We do seem to need more detail though to really help 
>>>>>>>>>> the
>>>>>>>>>> original OP.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Jason
>>>>>>>>>> Sent from my iPhone 5S
>>>>>>>>>> 
>>>>>>>>>>> On May 19, 2014, at 4:00 AM, Dave Cottlehuber <d...@jsonified.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> From: James Hoyt djnati...@gmail.com(mailto:djnati...@gmail.com)
>>>>>>>>>>> Reply: zfs-macos@googlegroups.com
>>>>>>>>>>> zfs-macos@googlegroups.com(mailto:zfs-macos@googlegroups.com)
>>>>>>>>>>> Date: 19. Mai 2014 at 02:27:36
>>>>>>>>>>> To: zfs-macos@googlegroups.com
>>>>>>>>>>> zfs-macos@googlegroups.com(mailto:zfs-macos@googlegroups.com)
>>>>>>>>>>> Subject: [zfs-macos] RAIDZ1 running slow =(
>>>>>>>>>>> 
>>>>>>>>>>>> So I setup a MacZFS RaidZ rather easily and was happy with
>>>>>>>>>>>> myself. I had four 3 TB internal SATA drives in a zpool giving me 
>>>>>>>>>>>> around 9
>>>>>>>>>>>> TB of space.
>>>>>>>>>>>> 
>>>>>>>>>>>> jamess-imac:~ sangie$ zpool status murr
>>>>>>>>>>>> pool: murr
>>>>>>>>>>>> state: ONLINE
>>>>>>>>>>>> scrub: none requested
>>>>>>>>>>>> config:
>>>>>>>>>>>> 
>>>>>>>>>>>> NAME STATE READ WRITE CKSUM
>>>>>>>>>>>> murr ONLINE 0 0 0
>>>>>>>>>>>> raidz1 ONLINE 0 0 0
>>>>>>>>>>>> disk3s2 ONLINE 0 0 0
>>>>>>>>>>>> disk1s2 ONLINE 0 0 0
>>>>>>>>>>>> disk2s2 ONLINE 0 0 0
>>>>>>>>>>>> disk4s2 ONLINE 0 0 0
>>>>>>>>>>>> 
>>>>>>>>>>>> errors: No known data errors
>>>>>>>>>>>> 
>>>>>>>>>>>> So I Filled it up with about 5 GBs of data, mainly images and
>>>>>>>>>>>> FLAC/music files and everything just drags on it. It takes a long 
>>>>>>>>>>>> time for
>>>>>>>>>>>> files to be listed in finder and when I try to save an image from 
>>>>>>>>>>>> Firefox,
>>>>>>>>>>>> it will just grind and grind while I try to navigate to a folder. 
>>>>>>>>>>>> I have
>>>>>>>>>>>> vmware Fusion setup on my SSD (my main Mac drive) and doing 
>>>>>>>>>>>> anything on my
>>>>>>>>>>>> zpool from Windows (like using MediaMonkey to organize FLAC files 
>>>>>>>>>>>> on it)
>>>>>>>>>>>> uses up 100% of the CPU, freezing up my computer until the moves 
>>>>>>>>>>>> are done,
>>>>>>>>>>>> even when moving around 30 files.
>>>>>>>>>>> 
>>>>>>>>>>> It’s not clear from this what your actual physical / virtual
>>>>>>>>>>> setup is. Are you booting to OSX, and running Windows in a VM? Is 
>>>>>>>>>>> the entire
>>>>>>>>>>> VM then living on the raidz pool?
>>>>>>>>>>> 
>>>>>>>>>>>> Is my zpool okay? What's going on? Is this type of slowness
>>>>>>>>>>>> normal or do I have a bad drive? How will MacZFS report to me if a 
>>>>>>>>>>>> drive in
>>>>>>>>>>>> the array goes bad? I installed SMARTReporter Lite and it shows 
>>>>>>>>>>>> all drives
>>>>>>>>>>>> as green. If I have some drives on SATA II and others on SATA III 
>>>>>>>>>>>> would that
>>>>>>>>>>>> affect anything?
>>>>>>>>>>>> 
>>>>>>>>>>>> If you want me to run any tests on it, I will do so gladly. Just
>>>>>>>>>>>> let me know.
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks!
>>>>>>>>>>> 
>>>>>>>>>>> I’ve seen precisely this sort of behaviour with vmware fusion
>>>>>>>>>>> when:
>>>>>>>>>>> 
>>>>>>>>>>> 1. my SSD was getting worn down (really, I trashed it in 1 year,
>>>>>>>>>>> it was the default apple one coming with early 2011 MBP)
>>>>>>>>>>> 2. the host OS & VM doesn’t have sufficient memory to run
>>>>>>>>>>> correctly without swapping
>>>>>>>>>>> 3. the additional memory within the VM is pulled from a disk swap
>>>>>>>>>>> file, which is by default in the same disk location as the VM itself
>>>>>>>>>>> 
>>>>>>>>>>> Anything less than 8GB of RAM is likely to be tight, VMs will of
>>>>>>>>>>> course make this more complicated. Some notes on
>>>>>>>>>>> http://artykul8.com/2012/06/vmware-performance-enhancing/ may help.
>>>>>>>>>>> 
>>>>>>>>>>> I found that my SSDs were being worn out with constant running of
>>>>>>>>>>> VMs; I use them heavily in my work. The solution I found was to get 
>>>>>>>>>>> max RAM
>>>>>>>>>>> in my laptop + imac (16 vs 32 respectively), make a zfs based 
>>>>>>>>>>> ramdisk with
>>>>>>>>>>> lz4 compression, and copy the entire VM into the ramdisk before 
>>>>>>>>>>> running it.
>>>>>>>>>>> The copy phase only takes a few seconds from SSD, and it gives me a 
>>>>>>>>>>> very
>>>>>>>>>>> nice way to “roll back” to the previous image when required. I can
>>>>>>>>>>> comfortably run Windows in a 20GiB ramdisk that fits inside a 10GiB 
>>>>>>>>>>> zpool
>>>>>>>>>>> with compression, even on the 16GiB laptop, and allocating 2GiB of 
>>>>>>>>>>> ram for
>>>>>>>>>>> the VM itself (10 + 2 for virtualisation & leave 4 for all of OSX 
>>>>>>>>>>> stuff).
>>>>>>>>>>> 
>>>>>>>>>>> Here’s the zsh functions I use for this.
>>>>>>>>>>> 
>>>>>>>>>>> # create a 1GiB ramdisk
>>>>>>>>>>> ramdisk-1g () {
>>>>>>>>>>> ramdisk-create 2097152
>>>>>>>>>>> }
>>>>>>>>>>> 
>>>>>>>>>>> # the generic function for the specific one above
>>>>>>>>>>> ramdisk-create () {
>>>>>>>>>>> diskutil eject /Volumes/ramdisk > /dev/null 2>&1
>>>>>>>>>>> diskutil erasevolume HFS+ 'ramdisk' `hdiutil attach -nomount
>>>>>>>>>>> ram://$1`
>>>>>>>>>>> cd /ramdisk
>>>>>>>>>>> }
>>>>>>>>>>> 
>>>>>>>>>>> # make a zpool backed ramdisk instead of the HFS+ ones above.
>>>>>>>>>>> Main advantage is compression. I get at least 2x more “disk” for 
>>>>>>>>>>> RAM with
>>>>>>>>>>> this approach.
>>>>>>>>>>> zdisk () {
>>>>>>>>>>> sudo zpool create -O compression=lz4 -fm /zram zram `hdiutil
>>>>>>>>>>> attach -nomount ram://20971520`
>>>>>>>>>>> sudo chown -R $USER /zram
>>>>>>>>>>> cd /zram
>>>>>>>>>>> }
>>>>>>>>>>> 
>>>>>>>>>>> # self explanatory
>>>>>>>>>>> zdisk-destroy () {
>>>>>>>>>>> sudo zpool export -f zram
>>>>>>>>>>> }
>>>>>>>>>>> 
>>>>>>>>>>> —
>>>>>>>>>>> Dave Cottlehuber
>>>>>>>>>>> d...@jsonified.com
>>>>>>>>>>> Sent from my Couch
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> --
>>>>>>>>>>> 
>>>>>>>>>>> ---
>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>> Google Groups "zfs-macos" group.
>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>>>> send an email to zfs-macos+unsubscr...@googlegroups.com.
>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> 
>>>>>>>>>> ---
>>>>>>>>>> You received this message because you are subscribed to a topic in
>>>>>>>>>> the Google Groups "zfs-macos" group.
>>>>>>>>>> To unsubscribe from this topic, visit
>>>>>>>>>> https://groups.google.com/d/topic/zfs-macos/78gD-0OzKMQ/unsubscribe.
>>>>>>>>>> To unsubscribe from this group and all its topics, send an email
>>>>>>>>>> to zfs-macos+unsubscr...@googlegroups.com.
>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> 
>>>>>>>>> ---
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>> Groups "zfs-macos" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>> send an email to zfs-macos+unsubscr...@googlegroups.com.
>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>> 
>>>>>>>> --
>>>>>>>> 
>>>>>>>> ---
>>>>>>>> You received this message because you are subscribed to a topic in
>>>>>>>> the Google Groups "zfs-macos" group.
>>>>>>>> To unsubscribe from this topic, visit
>>>>>>>> https://groups.google.com/d/topic/zfs-macos/78gD-0OzKMQ/unsubscribe.
>>>>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>>>>> zfs-macos+unsubscr...@googlegroups.com.
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>> 
>>>>>>> --
>>>>>>> 
>>>>>>> ---
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "zfs-macos" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to zfs-macos+unsubscr...@googlegroups.com.
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>> 
>>>>>> --
>>>>>> 
>>>>>> ---
>>>>>> You received this message because you are subscribed to a topic in the
>>>>>> Google Groups "zfs-macos" group.
>>>>>> To unsubscribe from this topic, visit
>>>>>> https://groups.google.com/d/topic/zfs-macos/78gD-0OzKMQ/unsubscribe.
>>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>>> zfs-macos+unsubscr...@googlegroups.com.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>> 
>>>>> --
>>>>> 
>>>>> ---
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "zfs-macos" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to zfs-macos+unsubscr...@googlegroups.com.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>> 
>>>> --
>>>> 
>>>> ---
>>>> You received this message because you are subscribed to a topic in the
>>>> Google Groups "zfs-macos" group.
>>>> To unsubscribe from this topic, visit
>>>> https://groups.google.com/d/topic/zfs-macos/78gD-0OzKMQ/unsubscribe.
>>>> To unsubscribe from this group and all its topics, send an email to
>>>> zfs-macos+unsubscr...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>> 
>>> --
>>> 
>>> ---
>>> You received this message because you are subscribed to the Google Groups
>>> "zfs-macos" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to zfs-macos+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>> 
>> 
>> --
>> 
>> ---
>> You received this message because you are subscribed to a topic in the
>> Google Groups "zfs-macos" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/zfs-macos/78gD-0OzKMQ/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> zfs-macos+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
> 
> 
>> On Tue, May 20, 2014 at 1:08 PM, Bjoern Kahl <googlelo...@bjoern-kahl.de> 
>> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>> 
>> 
>> Hi James,
>> 
>> pretty much all relevant things have already been said, so I can make
>> this short (well, didn't worked out).
>> 
>> MacZFS (stable version) comes with man pages.  A simple "man zpool"
>> should give you access to all the ZFS pool maintenance commands.
>> 
>> I virtually fell off my chair when I read your two statements
>> "I thought I need not backups" and "what is scrub, Does it do a data
>> wipe?"
>> 
>> As Jason said, ZFS is a wonderful piece of technology, but it is not
>> that kind of software one should use by just following some
>> setp-by-step guides.  It will sooner or later bite you.  We tried to
>> make it save and we tried to make it Mac friendly, but ZFS is
>> ultimately designed for big data centers and no interface magic can
>> really hide that fact.
>> 
>> 
>> Nevertheless,  to  answer  your  questions:
>> 
>> 
>> Scrub reads all data on a pool and verifies the checksums ZFS
>> maintains for each chunk of data stored in a pool.  Jason gave you
>> the commands in his other post.
>> 
>> If (big if) you have redundancy in your pool, that is a mirror or a
>> raidz, then and only then it can repair damaged data in the background.
>> 
>> It does so, by either getting a good copy from the other side(s) of
>> the mirror, or by combinatorial calculations from the raidz parity
>> stripes.
>> 
>> In a raidzX you can loose X drives without immediate data loss, in a
>> N-way mirror you can loose (N-1) drives without immediate data loss.
>> 
>> Note!  The keyword here is *immediate* data loss.  If you buy 3 drives
>> in a batch, and put these drives in a pool (mirror or raidz), then
>> these drives will experience similar workload under similar condition,
>> which significantly increases the likelihood to fail around the same
>> time.
>> 
>> Which means in a raidz1, you have a significant chance, that a second
>> drive will fail while you are in the process of replacing a first
>> failed drive.  The moment a second drive fails, your data is gone.
>> 
>> That is why you need backups.
>> 
>> I have personally seen this happen more than once, and switched to
>> always pairing drives from different manufactures and suppliers into
>> mirror pairs.  I say "and suppliers" to not have both drives
>> experience the same shuffles and drops to the ground while in
>> transportation.
>> 
>> And you need regular(!) scrubs, to find out that a drive is getting
>> weak before it fails completely, so you can replace it in time.
>> 
>> And one more word on replacing drives:
>> 
>> Once you have a drive failure, chances are you are in panic mode or at
>> least in a hurry to fix things, which means prone to make mistakes.
>> We are all just humans and do make mistakes.  So you should exercise a
>> drive replacement in advance.  Replacing a random drive on a redundant
>> pool using "zpool replace pool drive1 drive2" is supposed to be a safe
>> operation, so you can simply try it out.  The tricky part is how to
>> hookup the drives and identify the right drive, not the actual
>> replacement.
>> 
>> Using "zpool replace" instead of the sometimes suggested "zpool
>> attach" / "zpool detach" saves you from the all to common mistake to
>> say "zpool add" instead of "zpool attach", a mistake that would screw
>> up your pool layout and that can only be fixed by destroying and
>> recreating the pool.
>> 
>> 
>> Regarding the slowness:
>> 
>> Using 4k drives in a pool configure for 512b drives (the standard type
>> since hard drives were invented) will kill performance.
>> 
>> Using 512b drives in a pool configured for 4k drives does no harm,
>> except wasting a bit of space if you have many small files.
>> 
>> So I suggest to destroy and recreate the pool if your drives are 4k
>> (also called "enhanced format").  To configure a pool for 4k, you add
>> "-o ashift=12" the the "zpool create" command.  "zpool get all" should
>> tell you the current ashift value, which is 9 for 512b and 12 for 4k
>> 
>> :-) Exercise for the reader: Which ashift value to use for old style
>> 16k flash memory?  (Not that it would last long, but that's not the
>> point here.)
>> 
>> 
>> Regarding slow, long directories:
>> 
>> Another issue our colleagues working on the new MacZFS find out:
>> The Mac OSX kernel has a problem with caching really long directories,
>> because it can run out of some internal file resources (the famous
>> vnodes).  This hits ZFS especially hard due to the way it handles its
>> own short time locking and caching.
>> 
>> 
>> Best regards
>> 
>>    Björn
>> 
>> 
>> Am 20.05.14 18:09, schrieb James Hoyt:
>>> You have completely lost me at this point. You were rather
>>> condescending and not helpful. I was hoping for instructions on how
>>> to clean and scrub and saw none of that. At least point me to some
>>> proper links. I also don't know what a 4k drive is.
>>> 
>>> I carefully followed and read ALL the instructions and FAQ and
>>> Getting Started guide on maczfs.org. Please don't speak to me like
>>> I didn't do my research or follow the proper instructions.
>>> 
>>> - James
>>> 
>>> On Tue, May 20, 2014 at 9:24 AM, Jason Belec
>>> <jasonbe...@belecmartin.com> wrote:
>>>> OK, one thing, any indexing under that version of ZFS is going to
>>>> kill performance. Long standing issue.
>>>> 
>>>> No backups? Did you bump your noggin? With your current setup you
>>>> have improved your chances if your scrubbing regularly and if you
>>>> only lose a drive at anyone time. And adding backup will
>>>> drastically increase your chances.
>>>> 
>>>> Not understanding ZFS is a BIG reason to stop and re-evaluate
>>>> your priorities. It's amazing tech IF used properly.
>>>> 
>>>> For what it sounds like you want from ZFS you should use mirrors.
>>>> You can do 2 mirrors of 2 drives each stripped under ZFS. This
>>>> will increase the safety of your data. Even that should have a
>>>> back up drive you move key files or better yet 'snapshots' onto.
>>>> 
>>>> BUT you are going to have to understand ZFS to have any hope of
>>>> not drowning in a pool of tears at some point.
>>>> 
>>>> The new ZFS is under development but far more functional.
>>>> Eliminating many of the old version issues listed numerous times
>>>> throughout the forum. Either way you should ALWAYS understand the
>>>> tech you rely on. Period.
>>>> 
>>>> Please start learning with the word 'scrub' then the word
>>>> 'snapshot' and how to swap a failed drive and do it all. Before
>>>> committing your valuable data. Drives fail. Repeat. Drives fail.
>>>> Data must be restored at some point. ZFS is magical if you have
>>>> planned ahead. I have recovered data assumed totally lost, YMMV.
>>>> 
>>>> As for those drives are they 4k? If so you formatted your pool
>>>> incorrectly. I don't have any of those so I don't have notes.
>>>> Should be a simple Google search to find out. And the wiki has
>>>> the instructions on 4k drive setup.
>>>> 
>>>> Doing things right is what the wiki tries to help people with.
>>>> The forum allows you to search for other peoples heartbreak to
>>>> help prevent your own.  The wizards tracking this stuff have done
>>>> a wonderful job.
>>>> 
>>>> Hope this gets you rolling. I'd still check your cables as well.
>>>> Normally I attach a drive, build a pool, test a lot, destroy
>>>> pool. Add another drive. Repeat. Better safe than sorry.
>>>> Manufacturers are not safe guarding your data.
>>>> 
>>>> Jason Sent from my iPhone 5S
>>>> 
>>>>> On May 20, 2014, at 9:37 AM, James Hoyt <djnati...@gmail.com>
>>>>> wrote:
>>>>> 
>>>>> Thanks for the detailed reply.
>>>>> 
>>>>> The slow performance is only when I'm using the RAID array so I
>>>>> assume without it connected means I can't use it means there is
>>>>> no slow performance. I would love instructions on how to
>>>>> scrub/clean the pool. Does it do a data wipe?
>>>>> 
>>>>> I was trying to think of a good backup solution. I have over 3
>>>>> TBs of music in FLAC (lots of which I've paid for) and was
>>>>> hoping RAIDZ would take away the need for backups. I was
>>>>> thinking of buying a 4 TB drive and moving all my data on that
>>>>> and storing the drive offsite or something (in case of
>>>>> burglary, fires, etc). Having a single drive fail safe seems
>>>>> secure enough for me so I don't think incremental backups are
>>>>> needed.
>>>>> 
>>>>> As for running the latest beta ZFS, I didn't because the FAQ
>>>>> warned me not to. What are the differences? Would I have to
>>>>> format and rebuild the array?
>>>>> 
>>>>> The drives I have are four 3 TB Hitachi HDS723030BLE640.
>>>>> 
>>>>> I started navigating around my computer again, and the slowdown
>>>>> seems to be when going into folders with over 1000 files (for
>>>>> anything more it will take 1-3 minutes to just list the files
>>>>> in the directory). Also when I'm saving images from Firefox (no
>>>>> virtual machine running) it takes awhile to navigate the folder
>>>>> structure and sometimes not all the folders show, but they do
>>>>> in the Finder. So I wonder if this is an issue with programs
>>>>> not getting along with ZFS but the finder being fine with it.
>>>>> 
>>>>> Other things to note, I did disable Spotlight on the drive to
>>>>> make sure that isn't running, but I do have QuickSilver.
>>>>> Originally, I had QuickSilver indexing the drive, but the
>>>>> computer was practically unusable when it did that so I
>>>>> disabled that.
>>>>> 
>>>>> I look forward to any advice you guys may have.
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> James
>>>>> 
>>>>>> On Tue, May 20, 2014 at 6:14 AM, Jason Belec
>>>>>> <jasonbe...@belecmartin.com> wrote: OK, doesn't look like
>>>>>> RAM, processor etc., are the issue.... Let's work with that
>>>>>> in mind for now.
>>>>>> 
>>>>>> When the pool and the associated drives are not connected, is
>>>>>> the computer back to your expectation of normal? If so, you
>>>>>> have one or more bad cables, one or more bad drives, or a bit
>>>>>> of both, perhaps a bad or not quite capable power supply
>>>>>> (solves 90% of all issues I come across). Maybe even an issue
>>>>>> with the motherboard. Simplest thing, have you run a scrub on
>>>>>> this pool? Clean?
>>>>>> 
>>>>>> The type of drives you have is not an issue, the make and
>>>>>> known issues with said drives might be, but you didn't
>>>>>> provide that info.
>>>>>> 
>>>>>> Using a raidcard and macJournaled terms, thrown out will not
>>>>>> help you, your either ZFS or not. That said, you will not get
>>>>>> the same speed from ZFS as from other raid setups, but you
>>>>>> will get peace of mind on data integrity. I do hope you are
>>>>>> also backing up data from the pool as well or eventually you
>>>>>> will be in tears like so many others. A little forum
>>>>>> searching under old and new versions of mac zfs will be
>>>>>> helpful.
>>>>>> 
>>>>>> Since your getting started, once this is resolved it might be
>>>>>> better to build/run this under the latest (yes its in
>>>>>> development) Mac ZFS rather than the old tired version. It is
>>>>>> quite a bit different, modern and makes many things a lot
>>>>>> easier. (Insert legal disclaimer here) ;)
>>>>>> 
>>>>>> Interesting aside: Dave mentioned an interesting point about
>>>>>> wearing out SSDs, and I must admit I've had two such
>>>>>> occurrences but only with a hackintosh and only with less
>>>>>> than stellar drives. Seems that here around the mad science
>>>>>> lab Intel SSDs are the most reliable long term. I have two of
>>>>>> their originals still outlasting several other brands.
>>>>>> 
>>>>>> -- Jason Belec Sent from my iPad
>>>>>> 
>>>>>>> On May 19, 2014, at 10:05 AM, James Hoyt
>>>>>>> <djnati...@gmail.com> wrote:
>>>>>>> 
>>>>>>> Thanks for all the replies guys =D
>>>>>>> 
>>>>>>> Sorry for lack of information. I'm running a Hackintosh
>>>>>>> with a 256 GB SSD and I sometimes run Windows 8.1 in a
>>>>>>> virtual machine via VmWare Fusion. The virtual image file
>>>>>>> is also located on the SSD. The only files I have on my
>>>>>>> zpool are data files. I don't run an OS or VM image from
>>>>>>> it. I have 12 GBs of RAM and a four core i5 processor. On
>>>>>>> the VM, I dedicate 6 GBs of RAM and 2 cores to it. It
>>>>>>> should be noted that I experience the slow down even when
>>>>>>> vmware is off it's just the drives act the slowest when the
>>>>>>> VM is running.
>>>>>>> 
>>>>>>> As for how I created the zpool, I followed the Getting
>>>>>>> Started guide with
>>>>>>> 
>>>>>>> zpool create murr raidz disk3s2 disk1s2 disk2s2 disk4s2
>>>>>>> 
>>>>>>> Please help... I really hope I don't have to recreate it,
>>>>>>> but it's looking that way.
>>>>>>> 
>>>>>>> Would it be better if I bought a RAID card and use Mac OS
>>>>>>> Journaled? Cost is an issue... the other issue is these are
>>>>>>> regular desktop 7200 RPM drives.. not NAS drives.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> 
>>>>>>> James
>>>>>>> 
>>>>>>>> On Mon, May 19, 2014 at 7:43 AM, Jason Belec
>>>>>>>> <jasonbe...@belecmartin.com> wrote: Dave has posted some
>>>>>>>> good info. Reminds me why I prefer Virtualbox. ;) We do
>>>>>>>> seem to need more detail though to really help the
>>>>>>>> original OP.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Jason Sent from my iPhone 5S
>>>>>>>> 
>>>>>>>>> On May 19, 2014, at 4:00 AM, Dave Cottlehuber
>>>>>>>>> <d...@jsonified.com> wrote:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> From: James Hoyt
>>>>>>>>> djnati...@gmail.com(mailto:djnati...@gmail.com) Reply:
>>>>>>>>> zfs-macos@googlegroups.com
>>>>>>>>> zfs-macos@googlegroups.com(mailto:zfs-macos@googlegroups.com)
>> Date: 19. Mai 2014 at 02:27:36
>>>>>>>>> To: zfs-macos@googlegroups.com
>>>>>>>>> zfs-macos@googlegroups.com(mailto:zfs-macos@googlegroups.com)
>> Subject: [zfs-macos] RAIDZ1 running slow =(
>>>>>>>>> 
>>>>>>>>>> So I setup a MacZFS RaidZ rather easily and was happy
>>>>>>>>>> with myself. I had four 3 TB internal SATA drives in
>>>>>>>>>> a zpool giving me around 9 TB of space.
>>>>>>>>>> 
>>>>>>>>>> jamess-imac:~ sangie$ zpool status murr pool: murr
>>>>>>>>>> state: ONLINE scrub: none requested config:
>>>>>>>>>> 
>>>>>>>>>> NAME STATE READ WRITE CKSUM murr ONLINE 0 0 0 raidz1
>>>>>>>>>> ONLINE 0 0 0 disk3s2 ONLINE 0 0 0 disk1s2 ONLINE 0 0
>>>>>>>>>> 0 disk2s2 ONLINE 0 0 0 disk4s2 ONLINE 0 0 0
>>>>>>>>>> 
>>>>>>>>>> errors: No known data errors
>>>>>>>>>> 
>>>>>>>>>> So I Filled it up with about 5 GBs of data, mainly
>>>>>>>>>> images and FLAC/music files and everything just drags
>>>>>>>>>> on it. It takes a long time for files to be listed in
>>>>>>>>>> finder and when I try to save an image from Firefox,
>>>>>>>>>> it will just grind and grind while I try to navigate
>>>>>>>>>> to a folder. I have vmware Fusion setup on my SSD (my
>>>>>>>>>> main Mac drive) and doing anything on my zpool from
>>>>>>>>>> Windows (like using MediaMonkey to organize FLAC
>>>>>>>>>> files on it) uses up 100% of the CPU, freezing up my
>>>>>>>>>> computer until the moves are done, even when moving
>>>>>>>>>> around 30 files.
>>>>>>>>> 
>>>>>>>>> It’s not clear from this what your actual physical /
>>>>>>>>> virtual setup is. Are you booting to OSX, and running
>>>>>>>>> Windows in a VM? Is the entire VM then living on the
>>>>>>>>> raidz pool?
>>>>>>>>> 
>>>>>>>>>> Is my zpool okay? What's going on? Is this type of
>>>>>>>>>> slowness normal or do I have a bad drive? How will
>>>>>>>>>> MacZFS report to me if a drive in the array goes bad?
>>>>>>>>>> I installed SMARTReporter Lite and it shows all
>>>>>>>>>> drives as green. If I have some drives on SATA II and
>>>>>>>>>> others on SATA III would that affect anything?
>>>>>>>>>> 
>>>>>>>>>> If you want me to run any tests on it, I will do so
>>>>>>>>>> gladly. Just let me know.
>>>>>>>>>> 
>>>>>>>>>> Thanks!
>>>>>>>>> 
>>>>>>>>> I’ve seen precisely this sort of behaviour with vmware
>>>>>>>>> fusion when:
>>>>>>>>> 
>>>>>>>>> 1. my SSD was getting worn down (really, I trashed it
>>>>>>>>> in 1 year, it was the default apple one coming with
>>>>>>>>> early 2011 MBP) 2. the host OS & VM doesn’t have
>>>>>>>>> sufficient memory to run correctly without swapping 3.
>>>>>>>>> the additional memory within the VM is pulled from a
>>>>>>>>> disk swap file, which is by default in the same disk
>>>>>>>>> location as the VM itself
>>>>>>>>> 
>>>>>>>>> Anything less than 8GB of RAM is likely to be tight,
>>>>>>>>> VMs will of course make this more complicated. Some
>>>>>>>>> notes on
>>>>>>>>> http://artykul8.com/2012/06/vmware-performance-enhancing/
>>>>>>>>> may help.
>>>>>>>>> 
>>>>>>>>> I found that my SSDs were being worn out with constant
>>>>>>>>> running of VMs; I use them heavily in my work. The
>>>>>>>>> solution I found was to get max RAM in my laptop + imac
>>>>>>>>> (16 vs 32 respectively), make a zfs based ramdisk with
>>>>>>>>> lz4 compression, and copy the entire VM into the
>>>>>>>>> ramdisk before running it. The copy phase only takes a
>>>>>>>>> few seconds from SSD, and it gives me a very nice way
>>>>>>>>> to “roll back” to the previous image when required. I
>>>>>>>>> can comfortably run Windows in a 20GiB ramdisk that
>>>>>>>>> fits inside a 10GiB zpool with compression, even on the
>>>>>>>>> 16GiB laptop, and allocating 2GiB of ram for the VM
>>>>>>>>> itself (10 + 2 for virtualisation & leave 4 for all of
>>>>>>>>> OSX stuff).
>>>>>>>>> 
>>>>>>>>> Here’s the zsh functions I use for this.
>>>>>>>>> 
>>>>>>>>> # create a 1GiB ramdisk ramdisk-1g () { ramdisk-create
>>>>>>>>> 2097152 }
>>>>>>>>> 
>>>>>>>>> # the generic function for the specific one above
>>>>>>>>> ramdisk-create () { diskutil eject /Volumes/ramdisk >
>>>>>>>>> /dev/null 2>&1 diskutil erasevolume HFS+ 'ramdisk'
>>>>>>>>> `hdiutil attach -nomount ram://$1` cd /ramdisk }
>>>>>>>>> 
>>>>>>>>> # make a zpool backed ramdisk instead of the HFS+ ones
>>>>>>>>> above. Main advantage is compression. I get at least 2x
>>>>>>>>> more “disk” for RAM with this approach. zdisk () { sudo
>>>>>>>>> zpool create -O compression=lz4 -fm /zram zram `hdiutil
>>>>>>>>> attach -nomount ram://20971520` sudo chown -R $USER
>>>>>>>>> /zram cd /zram }
>>>>>>>>> 
>>>>>>>>> # self explanatory zdisk-destroy () { sudo zpool export
>>>>>>>>> -f zram }
>> 
>> - --
>> |     Bjoern Kahl   +++   Siegburg   +++    Germany     |
>> | "googlelogin@-my-domain-"   +++   www.bjoern-kahl.de  |
>> | Languages: German, English, Ancient Latin (a bit :-)) |
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1
>> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>> 
>> iQCVAgUBU3uaCVsDv2ib9OLFAQI9qwP9G8qRYdQD1w8q8nXCGKW23M9Ko8LjQq4n
>> N94yqQjzj7WbFYv6m1UMHl71EJkGuscyzKDzlOOqn3J5/hPsU2N12h0aN60qEgYJ
>> jJIIm+D5ujA+OqcnS2ChUYVSMgNyG19rd72zo+n5g/PXF/B2N+OVvZRNbs3d30Qa
>> D61uDWwvY5c=
>> =JP1Y
>> -----END PGP SIGNATURE-----
>> 
>> --
>> 
>> ---
>> You received this message because you are subscribed to a topic in the 
>> Google Groups "zfs-macos" group.
>> To unsubscribe from this topic, visit 
>> https://groups.google.com/d/topic/zfs-macos/78gD-0OzKMQ/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to 
>> zfs-macos+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
> 
> -- 
> 
> --- 
> You received this message because you are subscribed to the Google Groups 
> "zfs-macos" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to zfs-macos+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"zfs-macos" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to zfs-macos+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to