To specify the size of the disks GPFS uses one can use zvols. Then one can turn
on the zfs setting sync=always to perform safe writes since I'm using SATA
cards there is no BBU. In our testing, turning sync=on creates a 20%-30%
decrease in overall throughput on writes.
I do not have numbers of this setup vs hardware RAID6.
Thanks,
Chris
From: gpfsug-discuss-boun...@spectrumscale.org
[gpfsug-discuss-boun...@spectrumscale.org] on behalf of Jaime Pinto
[pi...@scinet.utoronto.ca]
Sent: Monday, June 13, 2016 12:02
To: gpfsug-discuss@spectrumscale.org
Subject: Re: [gpfsug-discuss] GPFS on ZFS! ... ?
As Marc, I also have questions related to performance.
Assuming we let ZFS take care of the underlying software raid, what
would be the difference between GPFS and Lustre for instance, for the
"parallel serving" at scale part of the file system. What would keep
GPFS from performing or functioning just as well?
Thanks
Jaime
Quoting "Marc A Kaplan" <makap...@us.ibm.com>:
> How do you set the size of a ZFS file that is simulating a GPFS disk? How
> do "tell" GPFS about that?
>
> How efficient is this layering, compared to just giving GPFS direct access
> to the same kind of LUNs that ZFS is using?
>
> Hmmm... to partially answer my question, I do something similar, but
> strictly for testing non-performance critical GPFS functions.
> On any file system one can:
>
> dd if=/dev/zero of=/fakedisks/d3 count=1 bs=1M seek=3000 # create a
> fake 3GB disk for GPFS
>
> Then use a GPFS nsd configuration record like this:
>
> %nsd: nsd=d3 device=/fakedisks/d3 usage=dataOnly pool=xtra
> servers=bog-xxx
>
> Which starts out as sparse and the filesystem will dynamically "grow" as
> GPFS writes to it...
>
> But I have no idea how well this will work for a critical "production"
> system...
>
> tx, marc kaplan.
>
>
>
> From: "Allen, Benjamin S." <bsal...@alcf.anl.gov>
> To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>
> Date: 06/13/2016 12:34 PM
> Subject:Re: [gpfsug-discuss] GPFS on ZFS?
> Sent by:gpfsug-discuss-boun...@spectrumscale.org
>
>
>
> Jaime,
>
> See
> https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm
> . An example I have for add /dev/nvme* devices:
>
> * GPFS doesn't know how that /dev/nvme* are valid block devices, use a
> user exit script to let it know about it
>
> cp /usr/lpp/mmfs/samples/nsddevices.sample /var/mmfs/etc/nsddevices
>
> * Edit /var/mmfs/etc/nsddevices, and add to linux section:
>
> if [[ $osName = Linux ]]
> then
> : # Add function to discover disks in the Linux environment.
> for dev in $( cat /proc/partitions | grep nvme | awk '{print $4}' )
> do
> echo $dev generic
> done
> fi
>
> * Copy edited nsddevices to the rest of the nodes at the same path
> for host in n01 n02 n03 n04; do
> scp /var/mmfs/etc/nsddevices ${host}:/var/mmfs/etc/nsddevices
> done
>
>
> Ben
>
>> On Jun 13, 2016, at 11:26 AM, Jaime Pinto <pi...@scinet.utoronto.ca>
> wrote:
>>
>> Hi Chris
>>
>> As I understand, GPFS likes to 'see' the block devices, even on a
> hardware raid solution such as DDN's.
>>
>> How is that accomplished when you use ZFS for software raid?
>> On page 4, I see this info, and I'm trying to interpret it:
>>
>> General Configuration
>> ...
>> * zvols
>> * nsddevices
>> - echo "zdX generic"
>>
>>
>> Thanks
>> Jaime
>>
>> Quoting "Hoffman, Christopher P" <cphof...@lanl.gov>:
>>
>>> Hi Jaime,
>>>
>>> What in particular would you like explained more? I'd be more than
> happy to discuss things further.
>>>
>>> Chris
>>>
>>> From: gpfsug-discuss-boun...@spectrumscale.org
> [gpfsug-discuss-boun...@spectrumscale.org] on behalf of Jaime Pinto
> [pi...@scinet.utoronto.ca]
>>> Sent: Monday, June 13, 2016 10:11
>>> To: gpfsug main discussion list
>>> Subject: Re: [gpfsug-discuss] GPFS on ZFS?
>>>
>>> I just came across this presentation on "GPFS with underlying ZFS
>>> block devices", by Christopher Hoffman, Los Alamos National Lab,
>>> although some of the
>>> implementation remains obscure.
>>>
>>> http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf
>>>
>>> It would be great to have more details, in particular the possibility
>>> of straight use