Re: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster

[Chris Scott]

Hi Fred

The imported filesystem has ~1.5M files that are migrated to Spectrum
Protect. Spot checking transparent and selective recalls of a handful of
files has been successful after associating them with their correct
Spectrum Protect server. They're all also backed up to primary and copy
pools in the Spectrum Protect server so having to do a restore instead of
recall if it wasn't working was an acceptable risk in favour of trying to
persist the GPFS 3.5 cluster on dying hardware and insecure OS, etc.

Cheers
Chris

On Mon, 1 Jun 2020 at 17:53, Frederick Stock  wrote:

> Chris, it was not clear to me if the file system you imported had files
> migrated to Spectrum Protect, that is stub files in GPFS.  If the file
> system does contain files migrated to Spectrum Protect with just a stub
> file in the file system, have you tried to recall any of them to see if
> that still works?
>
> Fred
> __
> Fred Stock | IBM Pittsburgh Lab | 720-430-8821
> sto...@us.ibm.com
>
>
>
> - Original message -
> From: Chris Scott 
> Sent by: gpfsug-discuss-boun...@spectrumscale.org
> To: gpfsug main discussion list 
> Cc:
> Subject: [EXTERNAL] Re: [gpfsug-discuss] Importing a Spectrum Scale a
> filesystem from 4.2.3 cluster to 5.0.4.3 cluster
> Date: Mon, Jun 1, 2020 9:14 AM
>
> Sounds like it would work fine.
>
> I recently exported a 3.5 version filesystem from a GPFS 3.5 cluster to a
> 'Scale cluster at 5.0.2.3 software and 5.0.2.0 cluster version. I
> concurrently mapped the NSDs to new NSD servers in the 'Scale cluster,
> mmexported the filesystem and changed the NSD servers configuration of the
> NSDs using the mmimportfs ChangeSpecFile. The original (creation)
> filesystem version of this filesystem is 3.2.1.5.
>
> To my pleasant surprise the filesystem mounted and worked fine while still
> at 3.5 filesystem version. Plan B would have been to "mmchfs 
> -V full" and then mmmount, but I was able to update the filesystem to
> 5.0.2.0 version while already mounted.
>
> This was further pleasantly successful as the filesystem in question is
> DMAPI-enabled, with the majority of the data on tape using Spectrum Protect
> for Space Management than the volume resident/pre-migrated on disk.
>
> The complexity is further compounded by this filesystem being associated
> to a different Spectrum Protect server than an existing DMAPI-enabled
> filesystem in the 'Scale cluster. Preparation of configs and subsequent
> commands to enable and use Spectrum Protect for Space Management
> multiserver for migration and backup all worked smoothly as per the docs.
>
> I was thus able to get rid of the GPFS 3.5 cluster on legacy hardware, OS,
> GPFS and homebrew CTDB SMB and NFS and retain the filesystem with its
> majority of tape-stored data on current hardware, OS and 'Scale/'Protect
> with CES SMB and NFS.
>
> The future objective remains to move all the data from this historical
> filesystem to a newer one to get the benefits of larger block and inode
> sizes, etc, although since the data is mostly dormant and kept for
> compliance/best-practice purposes, the main goal will be to head off
> original file system version 3.2 era going end of support.
>
> Cheers
> Chris
>
> On Thu, 28 May 2020 at 23:31, Prasad Surampudi <
> prasad.suramp...@theatsgroup.com> wrote:
>
> We have two scale clusters, cluster-A running version Scale 4.2.3 and
> RHEL6/7 and Cluster-B running Spectrum Scale  5.0.4 and RHEL 8.1. All the
> nodes in both Cluster-A and Cluster-B are direct attached and no NSD
> servers. We have our current filesystem gpfs_4 in Cluster-A  and new
> filesystem gpfs_5 in Cluster-B. We want to copy all our data from gpfs_4
> filesystem into gpfs_5 which has variable block size.  So, can we map NSDs
> of gpfs_4 to Cluster-B nodes and do a mmexportfs of gpfs_4 from Cluster-A
> and mmimportfs into Cluster-B so that we have both filesystems available on
> same node in Cluster-B for copying data across fiber channel? If
> mmexportfs/mmimportfs works, can we delete nodes from Cluster-A and add
> them to Cluster-B without upgrading RHEL or GPFS versions for now and  plan
> upgrading them at a later time?
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster

[Chris Scott]

Sounds like it would work fine.

I recently exported a 3.5 version filesystem from a GPFS 3.5 cluster to a
'Scale cluster at 5.0.2.3 software and 5.0.2.0 cluster version. I
concurrently mapped the NSDs to new NSD servers in the 'Scale cluster,
mmexported the filesystem and changed the NSD servers configuration of the
NSDs using the mmimportfs ChangeSpecFile. The original (creation)
filesystem version of this filesystem is 3.2.1.5.

To my pleasant surprise the filesystem mounted and worked fine while still
at 3.5 filesystem version. Plan B would have been to "mmchfs 
-V full" and then mmmount, but I was able to update the filesystem to
5.0.2.0 version while already mounted.

This was further pleasantly successful as the filesystem in question is
DMAPI-enabled, with the majority of the data on tape using Spectrum Protect
for Space Management than the volume resident/pre-migrated on disk.

The complexity is further compounded by this filesystem being associated to
a different Spectrum Protect server than an existing DMAPI-enabled
filesystem in the 'Scale cluster. Preparation of configs and subsequent
commands to enable and use Spectrum Protect for Space Management
multiserver for migration and backup all worked smoothly as per the docs.

I was thus able to get rid of the GPFS 3.5 cluster on legacy hardware, OS,
GPFS and homebrew CTDB SMB and NFS and retain the filesystem with its
majority of tape-stored data on current hardware, OS and 'Scale/'Protect
with CES SMB and NFS.

The future objective remains to move all the data from this historical
filesystem to a newer one to get the benefits of larger block and inode
sizes, etc, although since the data is mostly dormant and kept for
compliance/best-practice purposes, the main goal will be to head off
original file system version 3.2 era going end of support.

Cheers
Chris

On Thu, 28 May 2020 at 23:31, Prasad Surampudi <
prasad.suramp...@theatsgroup.com> wrote:

> We have two scale clusters, cluster-A running version Scale 4.2.3 and
> RHEL6/7 and Cluster-B running Spectrum Scale  5.0.4 and RHEL 8.1. All the
> nodes in both Cluster-A and Cluster-B are direct attached and no NSD
> servers. We have our current filesystem gpfs_4 in Cluster-A  and new
> filesystem gpfs_5 in Cluster-B. We want to copy all our data from gpfs_4
> filesystem into gpfs_5 which has variable block size.  So, can we map NSDs
> of gpfs_4 to Cluster-B nodes and do a mmexportfs of gpfs_4 from Cluster-A
> and mmimportfs into Cluster-B so that we have both filesystems available on
> same node in Cluster-B for copying data across fiber channel? If
> mmexportfs/mmimportfs works, can we delete nodes from Cluster-A and add
> them to Cluster-B without upgrading RHEL or GPFS versions for now and  plan
> upgrading them at a later time?
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

[gpfsug-discuss] Employment vacancy: Research Computing Specialist at University of Dundee, Scotland

[Chris Scott]

Hi All


This is an employment opportunity to work with Spectrum Scale and its
integration features with Spectrum Protect.


Please see or forward along the following link to an employment vacancy in
my team for a Research Computing Specialist here at the University of
Dundee:
https://www.jobs.dundee.ac.uk/fe/tpl_uod01.asp?s=4A515F4E5A565B1A=102157,4132345688=135360005=54715623342377=sepirmfpbecljxwhkl

Cheers
Chris

[image: University of Dundee shield logo] <http://uod.ac.uk/sig-home>

*Chris Scott*
Research Computing Manager
School of Life Sciences, UoD IT, University of Dundee
+44 (0)1382 386250 | c.y.sc...@dundee.ac.uk
[image: University of Dundee Facebook] <http://uod.ac.uk/sig-fb> [image:
University of Dundee Twitter] <http://uod.ac.uk/sig-tw> [image: University
of Dundee LinkedIn] <http://uod.ac.uk/sig-li> [image: University of Dundee
YouTube] <http://uod.ac.uk/sig-yt> [image: University of Dundee Instagram]
<http://uod.ac.uk/sig-ig> [image: University of Dundee Snapchat]
<http://uod.ac.uk/sig-sc>
*One of the world's top 200 universities* <http://uod.ac.uk/sig-strapline>
Times Higher Education World University Rankings 2018
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Recent Whitepapers from Yuri Volobuev

[Chris Scott]

Hear, hear. My appreciation to Yuri for his great contribution to the user
community of important details.

On 31 October 2016 at 16:19, Aaron Knister  wrote:

> Wow! That's a real shame. His ability to articulate his immense knowledge
> of GPFS was truly impressive.
>
> Sent from my iPhone
>
> On Oct 31, 2016, at 5:53 AM, Oesterlin, Robert <
> robert.oester...@nuance.com> wrote:
>
> For those of you who may not know, Yuri Volobuev has left IBM to pursue
> new challenges. Myself along with many others received so much help and
> keen insight from Yuri on all things GPFS.
>
>
>
> He will be missed.
>
>
>
>
>
> Bob Oesterlin
> Sr Principal Storage Engineer, Nuance
>
>
>
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size

[Chris Scott]

Hi Brian

This is exactly what I do with a SSD tier on top of 10K and 7.2K tiers.

HAWC is another recent option that might address Eric's requirement but
needs further consideration of the read requirements you want from the
small files.

Cheers
Chris

On 31 October 2016 at 17:23, Brian Marshall  wrote:

> When creating a "fast tier" storage pool in a filesystem is the normal
> case to create a placement policy that places all files in the fast tier
> and migrates out old and large files?
>
>
> Brian Marshall
>
> On Mon, Oct 31, 2016 at 1:20 PM, Jez Tucker  wrote:
>
>> Hey Bryan
>>
>>   There was a previous RFE for path placement from the UG, but Yuri told
>> me this was not techically possible as an inode has no knowledge about the
>> parent dentry.  (IIRC).You can see this in effect in the C API.  It is
>> possible to work this out at kernel level, but it's so costly that it
>> becomes non-viable at scale / performance.
>>
>> IBMers please chip in and expand if you will.
>>
>> Jez
>>
>>
>> On 31/10/16 17:09, Bryan Banister wrote:
>>
>> The File Placement Policy that you are trying to set cannot use the size
>> of the file to determine the placement of the file in a GPFS Storage Pool.
>> This is because GPFS has no idea what the file size will be when the file
>> is open()’d for writing.
>>
>>
>>
>> Hope that helps!
>>
>> -Bryan
>>
>>
>>
>> PS. I really wish that we could use a path for specifying data placement
>> in a GPFS Pool, and not just the file name, owner, etc.  I’ll submit a RFE
>> for this.
>>
>>
>>
>> *From:* gpfsug-discuss-boun...@spectrumscale.org [
>> mailto:gpfsug-discuss-boun...@spectrumscale.org
>> ] *On Behalf Of *J. Eric
>> Wonderley
>> *Sent:* Monday, October 31, 2016 11:53 AM
>> *To:* gpfsug main discussion list
>> *Subject:* [gpfsug-discuss] wanted...gpfs policy that places larger
>> files onto a pool based on size
>>
>>
>>
>> I wanted to do something like this...
>>
>>
>> [root@cl001 ~]# cat /opt/gpfs/home.ply
>> /*Failsafe migration of old small files back to spinning media
>> pool(fc_8T) */
>> RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70)
>> WEIGHT(ACCESS_TIME) TO POOL 'fc_8T'
>> /*Write files larger than 16MB to pool called "fc_8T" */
>> RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216
>> /*Move anything else to system pool */
>> RULE 'default' SET POOL 'system'
>>
>> Apparently there is no happiness using FILE_SIZE in a placement policy:
>> [root@cl001 ~]# mmchpolicy home /opt/gpfs/home.ply
>> Error while validating policy `home.ply': rc=22:
>> PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or variable
>> name in this context.
>> PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE
>> {{{FILE_SIZE}}}>16777216
>> runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home
>> /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply   failed.
>> mmchpolicy: Command failed. Examine previous error messages to determine
>> cause.
>>
>> Can anyone suggest a way to accomplish this using policy?
>>
>> --
>>
>> Note: This email is for the confidential use of the named addressee(s)
>> only and may contain proprietary, confidential or privileged information.
>> If you are not the intended recipient, you are hereby notified that any
>> review, dissemination or copying of this email is strictly prohibited, and
>> to please notify the sender immediately and destroy this email and any
>> attachments. Email transmission cannot be guaranteed to be secure or
>> error-free. The Company, therefore, does not make any guarantees as to the
>> completeness or accuracy of this email or any attachments. This email is
>> for informational purposes only and does not constitute a recommendation,
>> offer, request or solicitation of any kind to buy, sell, subscribe, redeem
>> or perform any type of transaction of a financial product.
>>
>>
>> ___
>> gpfsug-discuss mailing list
>> gpfsug-discuss at 
>> spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>>
>> ___
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>>
>
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] GPFS+TSM+HSM: staging vs. migration priority

[Chris Scott]

Not meaning to hjack the thread but while we're on the topic of transparent
recall: I'd like to be able to disable it such that I can use SS ILM
policies agreed with the data owners to "archive" their data and recover
disk space by migrating files to tape, marking them as immutable to defend
against accidental or malicious deletion and have some user interface that
would let them "retrieve" the data back to disk as writable again, subject
to sufficient free disk space and within any quota limits as applicable.

Cheers
Chris

On 9 March 2016 at 12:12, Jaime Pinto  wrote:

> Yes! A behavior along those lines would be desirable. Users understand
> very well what it means for a file system to be near full.
>
> Are there any customers already doing something similar?
>
> Thanks
> Jaime
>
> Quoting Dominic Mueller-Wicke01 :
>
>
>> Hi Jamie,
>>
>> I see. So, the recall-shutdown would be something for a short time period.
>> right? Just for the time it takes to migrate files out and free space. If
>> HSM would allow the recall-shutdown, the impact for the users would be
>> that
>> each access to migrated files would lead to an access denied error. Would
>> that be acceptable for the users?
>>
>> Greetings, Dominic.
>>
>>
>> __
>>
>> Dominic Mueller-Wicke | IBM Spectrum Protect Development | Technical Lead
>> |
>> +49 7034 64 32794 | dominic.muel...@de.ibm.com
>>
>> Vorsitzende des Aufsichtsrats: Martina Koederitz; Geschäftsführung: Dirk
>> Wittkopp
>> Sitz der Gesellschaft: Böblingen; Registergericht: Amtsgericht Stuttgart,
>> HRB 243294
>>
>>
>>
>> From:   Jaime Pinto 
>> To: Dominic Mueller-Wicke01/Germany/IBM@IBMDE
>> Cc: gpfsug-discuss@spectrumscale.org
>> Date:   08.03.2016 21:38
>> Subject:Re: [gpfsug-discuss] GPFS+TSM+HSM: staging vs. migration
>>
>> priority
>>
>>
>>
>> Thanks for the suggestions Dominic
>>
>> I remember playing around with premigrated files at the time, and that
>> was not satisfactory.
>>
>> What we are looking for is a configuration based parameter what will
>> basically break out of the "transparency for the user" mode, and not
>> perform any further recalling, period, if|when the file system
>> occupancy is above a certain threshold (98%). We would not mind if
>> instead gpfs would issue a preemptive "disk full" error message to any
>> user/app/job relying on those files to be recalled, so migration on
>> demand will have a chance to be performance. What we prefer is to swap
>> precedence, ie, any migration requests would be executed ahead of any
>> recalls, at least until a certain amount of free space on the file
>> system has been cleared.
>>
>> It's really important that this type of feature is present, for us to
>> reconsider the TSM version of HSM as a solution. It's not clear from
>> the manual that this can be accomplish in some fashion.
>>
>> Thanks
>> Jaime
>>
>> Quoting Dominic Mueller-Wicke01 :
>>
>>
>>>
>>> Hi,
>>>
>>> in all cases a recall request will be handled transparent for the user at
>>> the time a migrated files is accessed. This can't be prevented and has
>>>
>> two
>>
>>> down sides: a) the space used in the file system increases and b) random
>>> access to storage media in the Spectrum Protect server happens. With
>>>
>> newer
>>
>>> versions of Spectrum Protect for Space Management a so called tape
>>> optimized recall method is available that can reduce the impact to the
>>> system (especially Spectrum Protect server).
>>> If the problem was that the file system went out of space at the time the
>>> recalls came in I would recommend to reduce the threshold settings for
>>>
>> the
>>
>>> file system and increase the number of premigrated files. This will allow
>>> to free space very quickly if needed. If you didn't use the policy based
>>> threshold migration so far I recommend to use it. This method is
>>> significant faster compared to the classical HSM based threshold
>>>
>> migration
>>
>>> approach.
>>>
>>> Greetings, Dominic.
>>>
>>>
>>>
>> __
>>
>>
>>> Dominic Mueller-Wicke | IBM Spectrum Protect Development | Technical Lead
>>>
>> |
>>
>>> +49 7034 64 32794 | dominic.muel...@de.ibm.com
>>>
>>> Vorsitzende des Aufsichtsrats: Martina Koederitz; Geschäftsführung: Dirk
>>> Wittkopp
>>> Sitz der Gesellschaft: Böblingen; Registergericht: Amtsgericht Stuttgart,
>>> HRB 243294
>>> - Forwarded by Dominic Mueller-Wicke01/Germany/IBM on 08.03.2016
>>>
>> 18:21
>>
>>> -
>>>
>>> From:Jaime Pinto 
>>> To:  gpfsug main discussion list <
>>> gpfsug-discuss@spectrumscale.org>
>>> Date:08.03.2016 17:36
>>> Subject: [gpfsug-discuss] 

Re: [gpfsug-discuss] GPFS+TSM+HSM: migration via GPFS policy scripts

[Chris Scott]

To add a customer data point, I followed that guide using GPFS 3.4 and TSM
6.4 with HSM and it's been working perfectly since then. I was even able to
remove dsmscoutd online, node-at-a-time back when I made the transition.
The performance change was revolutionary and so is the file selection.

We have large filesystems with millions of files, changing often, that TSM
incremental scan wouldn't cope with and Spectrum Scale 4.1.1 and Spectrum
Protect 7.1.3 using mmbackup as described in the SS 4.1.1 manual, creating
a snapshot for mmbackup also works perfectly for backup.

Cheers
Chris

On 8 March 2016 at 17:45, Dominic Mueller-Wicke01 <
dominic.muel...@de.ibm.com> wrote:

> Hi,
>
> please have a look at this document:
> http://www-01.ibm.com/support/docview.wss?uid=swg27018848
> It describe the how-to setup and provides some hints and tips for
> migration policies.
>
> Greetings, Dominic.
>
>
> __
> Dominic Mueller-Wicke | IBM Spectrum Protect Development | Technical Lead
> | +49 7034 64 32794 | dominic.muel...@de.ibm.com
>
> Vorsitzende des Aufsichtsrats: Martina Koederitz; Geschäftsführung: Dirk
> Wittkopp
> Sitz der Gesellschaft: Böblingen; Registergericht: Amtsgericht Stuttgart,
> HRB 243294
>
>
>
>
>
> For the new Spectrum Suite of products, are there specific references
> with examples on how to set up gpfs policy rules to integrate TSM so
> substantially improve the migration performance of HSM?
>
> The reason I ask is because I've been reading manuals with 200+ pages
> where it's very clear this is possible to be accomplished, by builtin
> lists and feeding those to TSM, however some of the examples and rules
> are presented out of context, and not integrated onto a single
> self-contained document. The GPFS past has it own set of manuals, but
> so do TSM and HSM.
>
> For those of you already doing it, what has been your experience, what
> are the tricks (where can I read about them), how the addition of
> multiple nodes to the working pool is performing?
>
> Thanks
> Jaime
>
>
>
> ---
> Jaime Pinto
> SciNet HPC Consortium  - Compute/Calcul Canada
> www.scinet.utoronto.ca - www.computecanada.org
> University of Toronto
> 256 McCaul Street, Room 235
> Toronto, ON, M5T1W5
> P: 416-978-2755
> C: 416-505-1477
>
> 
> This message was sent using IMP at SciNet Consortium, University of
> Toronto.
>
>
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Small cluster

[Chris Scott]

My fantasy solution is 2 servers and a SAS disk shelf from my adopted,
cheap x86 vendor running IBM Spectrum Scale with GNR as software only,
doing concurrent, supported GNR and CES with maybe an advisory on the
performance requirements of such and suggestions on scale out approaches :)

Cheers
Chris

On 7 March 2016 at 21:10, mark.b...@siriuscom.com 
wrote:

> Thanks Yuri, this solidifies some of the conclusions I’ve drawn from this
> conversation.  Thank you all for your responses.  This is a great forum
> filled with very knowledgeable folks.
>
> Mark
>
> From:  on behalf of Yuri L
> Volobuev 
> Reply-To: gpfsug main discussion list 
> Date: Monday, March 7, 2016 at 2:58 PM
> To: gpfsug main discussion list 
> Subject: Re: [gpfsug-discuss] Small cluster
>
> This use case is a good example of how it's hard to optimize across
> multiple criteria.
>
> If you want a pre-packaged solution that's proven and easy to manage,
> StorWize V7000 Unified is the ticket. Design-wise, it's as good a fit for
> your requirements as such things get. Price may be an issue though, as
> usual.
>
> If you're OK with rolling your own complex solution, my recommendation
> would be to use a low-end shared (twin-tailed, via SAS or FC SAN) external
> disk solution, with 2-3 GPFS nodes accessing the disks directly, i.e. via
> the local block device interface. This avoids the pitfalls of data/metadata
> replication, and offers a decent blend of performance, fault tolerance, and
> disk management. You can use disk-based quorum if going with 2 nodes, or
> traditional node majority quorum if using 3 nodes, either way would work.
> There's no need to do any separation of roles (CES, quorum, managers, etc),
> provided the nodes are adequately provisioned with memory and aren't
> routinely overloaded, in which case you just need to add more nodes instead
> of partitioning what you have.
>
> Using internal disks and relying on GPFS data/metadata replication, with
> or without FPO, would mean taking the hard road. You may be able to spend
> the least on hardware in such a config (although the 33% disk utilization
> rate for triplication makes this less clear, if capacity is an issue), but
> the operational challenges are going to be substantial. This would be a
> viable config, but there are unavoidable tradeoffs caused by replication:
> (1) writes are very expensive, which limits the overall cluster capability
> for non-read-only workloads, (2) node and disk failures require a round of
> re-replication, or "re-protection", which takes time and bandwidth,
> limiting the overall capability further, (3) disk management can be a
> challenge, as there's no software/hardware component to assist with
> identifying failing/failed disks. As far as not going off the beaten path,
> this is not it... Exporting protocols from a small triplicated file system
> is not a typical mode of deployment of Spectrum Scale, you'd be blazing
> some new trails.
>
> As stated already in several responses, there's no hard requirement that
> CES Protocol nodes must be entirely separate from any other roles in the
> general Spectrum Scale deployment scenario. IBM expressly disallows
> co-locating Protocol nodes with ESS servers, due to resource consumption
> complications, but for non-ESS cases it's merely a recommendation to run
> Protocols on nodes that are not otherwise encumbered by having to provide
> other services. Of course, the config that's the best for performance is
> not the cheapest. CES doesn't reboot nodes to recover from NFS problems,
> unlike cNFS (which has to, given its use of kernel NFS stack). Of course, a
> complex software stack is a complex software stack, so there's greater
> potential for things to go sideways, in particular due to the lack of
> resources.
>
> FPO vs plain replication: this only matters if you have apps that are
> capable of exploiting data locality. FPO changes the way GPFS stripes data
> across disks. Without FPO, GPFS does traditional wide striping of blocks
> across all disks in a given storage pool. When FPO is in use, data in large
> files is divided in large (e.g. 1G) chunks, and there's a node that holds
> an entire chunk on its internal disks. An application that knows how to
> query data block layout of a given file can then schedule the job that
> needs to read from this chunk on the node that holds a local copy. This
> makes a lot of sense for integrated data analytics workloads, a la Map
> Reduce with Hadoop, but doesn't make sense for generic apps like Samba.
>
> I'm not sure what language in the FAQ creates the impression that the SAN
> deployment model is somehow incompatible with running Procotol services.
> This is perfectly fine.
>
> yuri
>
> [image: Inactive hide details for Jan-Frode Myklebust ---03/06/2016
> 10:12:07 PM---I agree, but would also