I'm a big fan of ZFS, we have long used it behind Galaxy Main. Some of our
older servers are (still) Solaris, and the newest is FreeBSD.
I've lately been using SmartOS for virtualization and while it has a drawback
as a fileserver (since currently the nfs server can only run in the global
zone, which is not ideal on SmartOS), there are other illumos derivatives that
would probably be great for this task (e.g. OmniOS). Native ZFS in the OS in
which it is developed is a win for me, especially when you are serving via
simple NFS. For more complex network filesystems, Linux is probably preferable.
I considered a separate ZIL and L2ARC for the latest ZFS server, but DTrace
revealed that I probably would not see much of a performance benefit with our
usage patterns. The memory usage you're seeing is to be expected - it will
pretty much consume whatever is available for caching, but it's available to be
freed if needed for something else.
I wouldn't suggest rsync for performance testing. I typically do things like
timed writing of blocks read from /dev/zero using dd, so that the source
filesystem and checksumming algorithm can be taken out of the equation. And
dedup/compression will of course cause a significant write penalty. If you can
suffer the decreased space optimization, lzjb performs significantly better
than gzip. gzip-1 is a nice compromise between the default gzip level and
lzjb, as well.
On Sep 11, 2013, at 3:45 AM, Joachim Jacob | VIB | wrote:
> Thank you all for the reactions.
> Some details about my current ZFS and Galaxy setup:
> - Galaxy runs as a single virtual machine, with currently 20 cores, 80GB RAM.
> Will be 32 cores and about 160GB RAM soon.
> - The postgres database is on the virtual machine itself.
> - The 'files' and 'job_working_dir' are on an NFS exported directory, hosted
> on the host machine of the guest.
> - The NFS exported directory is an raidz1 dataset.
> - The raidsz1 runs on 7 550GB SAS disks, which are (unfortunately) controlled
> by a RAID hardware controller, but passed as RAID0 (JBOD not available). So
> raidz1 runs on 7 RAID0 disks (with settings in the hardware controller PERC
> H700: no read ahead, write through, 8 KB stripe size, disk cache policy
> - Compression and deduplication is enabled.
> - The directory on which the zfs dataset is mounted, is exported using the
> native linux NFS daemon to the Galaxy virtual machine. The 'zfs sharenfs' did
> not work (ownerships not set correctly - perhaps need some more
> investigation, but I found several times reports about sharenfs option in ZFS
> in linux is not behaving well...).
> The numbers:
> - my initial files database (ext4 on RAID5) is 3.0TB in size. On ZFS, with
> compression and deduplication, this database is *1.8TB *(-40%).
> - Did not yet provide a SLOG to host the ZIL and a L2ARC, since I have a
> cleared picture about the performance I can get. Would you advise preference
> over ZIL on a SLOG, or go for a SSD to host the L2ARC.
> - The cost for this performance of ZFS in storage is RAM: currently
> continuously using*284GB RAM* for ZFS!
> - The write and read speed is from the Galaxy VM over NFS is *~40MB/s and
> ~100MB/s* (tested by simply copying over rsync - I still need to check the
> presentation and scripts of Anne Black-Ziegelbein). This is a 66% decrease in
> previously achieved write and read speed (ext4 on hardware RAID5), but I feel
> that the benefits (deduplication, backing up via snapshots, data integrity,)
> outweigh this IO performance.
> (I am setting this ZFS up on a new server (well, actually 2 years old now,
> has served on another project well))
> Currently our Galaxy uses this zfs with success!
> For your interest, my settings on the 'galaxydb' zfs tank below. (I was
> wondering if here some more wizardry can be applied).
> [root@r910bits ~]# *zfs get all tank/galaxydb*
> NAME PROPERTY VALUE SOURCE
> tank/galaxydb type filesystem -
> tank/galaxydb creation Mon Sep 9 12:44 2013 -
> tank/galaxydb used 1.81T -
> tank/galaxydb available 1.66T -
> tank/galaxydb referenced 1.81T -
> tank/galaxydb compressratio 1.66x -
> tank/galaxydb mounted yes -
> tank/galaxydb quota none default
> tank/galaxydb reservation none default
> tank/galaxydb recordsize 128K default
> tank/galaxydb mountpoint /mnt/galaxydb local
> tank/galaxydb sharenfs rw=@galaxy local
> tank/galaxydb checksum on default
> tank/galaxydb compression lzjb local
> tank/galaxydb atime on default
> tank/galaxydb devices on default
> tank/galaxydb exec on default
> tank/galaxydb setuid on default
> tank/galaxydb readonly off default
> tank/galaxydb zoned off default
> tank/galaxydb snapdir hidden default
> tank/galaxydb aclinherit restricted default
> tank/galaxydb canmount on default
> tank/galaxydb xattr on default
> tank/galaxydb copies 1 default
> tank/galaxydb version 5 -
> tank/galaxydb utf8only off -
> tank/galaxydb normalization none -
> tank/galaxydb casesensitivity sensitive -
> tank/galaxydb vscan off default
> tank/galaxydb nbmand off default
> tank/galaxydb sharesmb off default
> tank/galaxydb refquota none default
> tank/galaxydb refreservation none default
> tank/galaxydb primarycache all default
> tank/galaxydb secondarycache all default
> tank/galaxydb usedbysnapshots 0 -
> tank/galaxydb usedbydataset 1.81T -
> tank/galaxydb usedbychildren 0 -
> tank/galaxydb usedbyrefreservation 0 -
> tank/galaxydb logbias latency default
> tank/galaxydb dedup on local
> tank/galaxydb mlslabel none default
> tank/galaxydb sync standard default
> tank/galaxydb refcompressratio 1.66x -
> tank/galaxydb written 1.81T -
> tank/galaxydb snapdev hidden default
> Joachim Jacob
> Contact details: http://www.bits.vib.be/index.php/about/80-team
> On 09/10/2013 11:29 PM, Guest, Simon wrote:
>> Hi Joachim,
>> At AgResearch we are using ZFS for our HPC storage, which is used by our
>> internal Galaxy instance. Currently we are running on FreeNAS (FreeBSD
>> derivative), but we are in transition to ZFS on Linux. We export the
>> filesystem over NFS (10Gb Ethernet), but not the database (PostgreSQL). In
>> general, you want block storage for a database, so I suggest you look for a
>> solution other than NFS to host that.
>> Our experience with ZFS has been very positive. However, FreeNAS is not
>> really suited to our needs - it's more of a storage appliance, probably
>> great for a home NAS. Hence the planned transition.
>> I strongly recommend you follow the discussion on the ZFS discuss mailing
>> list. There's a lot to learn about ZFS configuration, much more than you
>> will glean from few posts here.
>>> -----Original Message-----
>>> From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-
>>> boun...@lists.bx.psu.edu] On Behalf Of Joachim Jacob | VIB |
>>> Sent: Tuesday, 10 September 2013 10:29 p.m.
>>> To: firstname.lastname@example.org
>>> Subject: [galaxy-dev] ZFS storage recommendations
>>> Hi all,
>>> I am performing some tests to move my galaxy database to ZFS. Does anybody
>>> have experience with ZFS on linux, and some recommendations/experiences to
>>> optimize performance? The purpose is to share the database over NFS to the
>>> Galaxy VM.
>>> Joachim Jacob
>>> Contact details: http://www.bits.vib.be/index.php/about/80-team
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client. To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>> To search Galaxy mailing lists use the unified search at:
>> Attention: The information contained in this message and/or attachments
>> from AgResearch Limited is intended only for the persons or entities
>> to which it is addressed and may contain confidential and/or privileged
>> material. Any review, retransmission, dissemination or other use of, or
>> taking of any action in reliance upon, this information by persons or
>> entities other than the intended recipients is prohibited by AgResearch
>> Limited. If you have received this message in error, please notify the
>> sender immediately.
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> To search Galaxy mailing lists use the unified search at:
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
To search Galaxy mailing lists use the unified search at: