Re: [gpfsug-discuss] GPFS heartbeat network specifications and resilience

[Brian Marshall]

Sort of trailing on this thread - Is a bonded active-active 10gig ethernet
network enough bandwidth to run data and heartbeat/admin on the same
network?  I assume it comes down to a question of latency and congestion
but would like to hear others' stories.

Is anyone doing anything fancy with QOS to make sure admin/heartbeat
traffic is not delayed?

All of our current clusters use Infiniband for data and mgt traffic, but we
are building a cluster that has dual 10gigE to each compute node. The NSD
servers have 40gigE connections to the core network where 10gigE switches
uplink.

On Fri, Jul 22, 2016 at 4:57 AM, Ashish Thandavan <
ashish.thanda...@cs.ox.ac.uk> wrote:

> Hi Richard,
>
> Thank you, that is very good to know!
>
> Regards,
> Ash
>
>
> On 22/07/16 09:36, Sobey, Richard A wrote:
>
>> Hi Ash
>>
>> Our ifcfg files for the bonded interfaces (this applies to GPFS, data and
>> mgmt networks) are set to mode1:
>>
>> BONDING_OPTS="mode=1 miimon=200"
>>
>> If we have ever had a network outage on the ports for these interfaces,
>> apart from pulling a cable for testing when they went in, then I guess we
>> have it setup right as we've never noticed an issue. The specific mode1 was
>> asked for by our networks team.
>>
>> Richard
>>
>> -Original Message-
>> From: gpfsug-discuss-boun...@spectrumscale.org [mailto:
>> gpfsug-discuss-boun...@spectrumscale.org] On Behalf Of Ashish Thandavan
>> Sent: 21 July 2016 11:26
>> To: gpfsug-discuss@spectrumscale.org
>> Subject: [gpfsug-discuss] GPFS heartbeat network specifications and
>> resilience
>>
>> Dear all,
>>
>> Please could anyone be able to point me at specifications required for
>> the GPFS heartbeat network? Are there any figures for latency, jitter, etc
>> that one should be aware of?
>>
>> I also have a related question about resilience. Our three GPFS NSD
>> servers utilize a single network port on each server and communicate
>> heartbeat traffic over a private VLAN. We are looking at improving the
>> resilience of this setup by adding an additional network link on each
>> server (going to a different member of a pair of stacked switches than the
>> existing one) and running the heartbeat network over bonded interfaces on
>> the three servers. Are there any recommendations as to which network
>> bonding type to use?
>>
>> Based on the name alone, Mode 1 (active-backup) appears to be the ideal
>> choice, and I believe the switches do not need any special configuration.
>> However, it has been suggested that Mode 4 (802.3ad) or LACP bonding might
>> be the way to go; this aggregates the two ports and does require the
>> relevant switch ports to be configured to support this.
>> Is there a recommended bonding mode?
>>
>> If anyone here currently uses bonded interfaces for their GPFS heartbeat
>> traffic, may I ask what type of bond have you configured? Have you had any
>> problems with the setup? And more importantly, has it been of use in
>> keeping the cluster up and running in the scenario of one network link
>> going down?
>>
>> Thank you,
>>
>> Regards,
>> Ash
>>
>>
>>
>> --
>> -
>> Ashish Thandavan
>>
>> UNIX Support Computing Officer
>> Department of Computer Science
>> University of Oxford
>> Wolfson Building
>> Parks Road
>> Oxford OX1 3QD
>>
>> Phone: 01865 610733
>> Email: ashish.thanda...@cs.ox.ac.uk
>>
>> ___
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>> ___
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>
> --
> -
> Ashish Thandavan
>
> UNIX Support Computing Officer
> Department of Computer Science
> University of Oxford
> Wolfson Building
> Parks Road
> Oxford OX1 3QD
>
> Phone: 01865 610733
> Email: ashish.thanda...@cs.ox.ac.uk
>
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

[gpfsug-discuss] GPFS on Broadwell processor

[Brian Marshall]

All,

Is there anything special (BIOS option / kernel option) that needs to be
done when running GPFS on a Broadwell powered NSD server?


Thank you,
Brian
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] SSD LUN setup

[Brian Marshall]

That's very good advice.  In my specific case, I am looking at lowlevel
setup of the NSDs in a SSD storage pool with metadata stored elsewhere (on
another SSD system).  I am wondering if stuff like SSD pagepool size comes
into play or if I just look at the segment size from the storage enclosure
RAID controller.

It sounds like SSDs should be used just like HDDs: group them into RAID6
LUNs.  Write endurance is good enough now that longevity is not a problem
and there are plenty of IOPs to do parity work.  Does this sound right?
Anyone doing anything else?

Brian

On Sun, Jul 17, 2016 at 9:05 AM, Oesterlin, Robert <
robert.oester...@nuance.com> wrote:

> Thinly provisioned (compressed) metadata volumes is unsupported according
> to IBM. See the GPFS FAQ here, question 4.12:
>
>
>
> "Placing GPFS metadata on an NSD backed by a thinly provisioned volume is
> dangerous and unsupported."
>
>
>
> http://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html
>
>
>
> Bob Oesterlin
> Sr Storage Engineer, Nuance HPC Grid
> 507-269-0413
>
>
>
>
>
> *From: *<gpfsug-discuss-boun...@spectrumscale.org> on behalf of Brian
> Marshall <mimar...@vt.edu>
> *Reply-To: *gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>
> *Date: *Saturday, July 16, 2016 at 9:56 PM
> *To: *"gpfsug-discuss@spectrumscale.org" <gpfsug-discuss@spectrumscale.org
> >
> *Subject: *[EXTERNAL] [gpfsug-discuss] SSD LUN setup
>
>
>
> I have read about other products doing RAID1 with deduplication and
> compression to take less than the 50% capacity hit.
>
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

[gpfsug-discuss] segment size and sub-block size

[Brian Marshall]

All,

When picking blockSize and segmentSize on RAID6 8+2 LUNs, I have see 2
optimal theories.


1) Make blockSize = # Data Disks * segmentSize
e.g. in the RAID6 8+2 case,  8 MB blockSize = 8 * 1 MB segmentSize

This makes sense to me  as every GPFS block write is a full stripe write

2) Make blockSize = 32 (number sub blocks) * segmentSize; also make sure
the blockSize is a multiple of #data disks * segmentSize

I don't know enough about GPFS to know how subblocks interact and what
tradeoffs this makes.

Can someone explain (or point to a doc) about sub block mechanics and when
to optimize for that?

Thank you,
Brian Marshall
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

[gpfsug-discuss] Mounting GPFS data on OpenStack VM

[Brian Marshall]

UG,

I have a GPFS filesystem.

I have a OpenStack private cloud.

What is the best way for Nova Compute VMs to have access to data inside the
GPFS filesystem?

1)Should VMs mount GPFS directly with a GPFS client?
2) Should the hypervisor mount GPFS and share to nova computes?
3) Should I create GPFS protocol servers that allow nova computes to mount
of NFS?

All advice is welcome.


Best,
Brian Marshall
Virginia Tech
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Mounting GPFS data on OpenStack VM

[Brian Marshall]

To answer some more questions:

What sort of workload will your Nova VM's be running?
This is largely TBD but we anticipate webapps and other non-batch ways of
interacting with and post processing data that has been computed on HPC
batch systems.  For example a user might host a website that allows users
to view pieces of a large data set and do some processing in private cloud
or kick off larger jobs on HPC clusters

How many VM's are you running?
This work is still in the design / build phase.  We have 48 servers slated
for the project.  At max maybe 500 VMs; again this is a pretty wild
estimate.  This is a new service we are looking to provide

What is your Network interconnect between the Scale Storage cluster and the
Nova Compute cluster
Each nova node has a dual 10gigE connection to switches that uplink to our
core 40 gigE switches were NSD Servers are directly connectly.

The information so far has been awesome.  Thanks everyone.  I am definitely
leaning towards option #3 of creating protocol servers.  Are there any
design/build white papers targetting the virutalization use case?

Thanks,
Brian

On Tue, Jan 17, 2017 at 5:55 PM, Andrew Beattie <abeat...@au1.ibm.com>
wrote:

> HI Brian,
>
>
> Couple of questions for you:
>
> What sort of workload will your Nova VM's be running?
> How many VM's are you running?
> What is your Network interconnect between the Scale Storage cluster and
> the Nova Compute cluster
>
> I have cc'd Jake Carrol from University of Queensland in on the email as I
> know they have done some basic performance testing using Scale to provide
> storage to Openstack.
> One of the issues that they found was the Openstack network translation
> was a performance limiting factor.
>
> I think from memory the best performance scenario they had was, when they
> installed the scale client locally into the virtual machines
>
>
> Andrew Beattie
> Software Defined Storage  - IT Specialist
> Phone: 614-2133-7927
> E-mail: abeat...@au1.ibm.com
>
>
>
> - Original message -
> From: Brian Marshall <mimar...@vt.edu>
> Sent by: gpfsug-discuss-boun...@spectrumscale.org
> To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>
> Cc:
> Subject: [gpfsug-discuss] Mounting GPFS data on OpenStack VM
> Date: Wed, Jan 18, 2017 7:51 AM
>
> UG,
>
> I have a GPFS filesystem.
>
> I have a OpenStack private cloud.
>
> What is the best way for Nova Compute VMs to have access to data inside
> the GPFS filesystem?
>
> 1)Should VMs mount GPFS directly with a GPFS client?
> 2) Should the hypervisor mount GPFS and share to nova computes?
> 3) Should I create GPFS protocol servers that allow nova computes to mount
> of NFS?
>
> All advice is welcome.
>
>
> Best,
> Brian Marshall
> Virginia Tech
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
>
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] rmda errors scatter thru gpfs logs

[Brian Marshall]

As background, we recently upgraded GPFS from 4.2.0 to 4.2.1  and updated
the Mellanox OFED on our compute cluster to allow it to move from CentOS
7.1 to 7.2

We do some transient warnings from the Mellanox switch gear about various
port counters that we are tracking down with them.

Jobs and filesystem seem stable, but the logs are concerning.

On Wed, Jan 18, 2017 at 10:22 AM, Aaron Knister 
wrote:

> I'm curious about this too. We see these messages sometimes when things
> have gone horribly wrong but also sometimes during recovery events. Here's
> a recent one:
>
> loremds20 (manager/nsd node):
> Mon Jan 16 14:19:02.048 2017: [E] VERBS RDMA rdma read error
> IBV_WC_REM_ACCESS_ERR to 10.101.11.6 (lorej006) on mlx5_0 port 1 fabnum 3
> vendor_err 136
> Mon Jan 16 14:19:02.049 2017: [E] VERBS RDMA closed connection to
> 10.101.11.6 (lorej006) on mlx5_0 port 1 fabnum 3 due to RDMA read error
> IBV_WC_REM_ACCESS_ERR index 11
>
> lorej006 (client):
> Mon Jan 16 14:19:01.990 2017: [N] VERBS RDMA closed connection to
> 10.101.53.18 (loremds18) on mlx5_0 port 1 fabnum 3 index 2
> Mon Jan 16 14:19:01.995 2017: [N] VERBS RDMA closed connection to
> 10.101.53.19 (loremds19) on mlx5_0 port 1 fabnum 3 index 0
> Mon Jan 16 14:19:01.997 2017: [I] Recovering nodes: 10.101.53.18
> 10.101.53.19
> Mon Jan 16 14:19:02.047 2017: [W] VERBS RDMA async event
> IBV_EVENT_QP_ACCESS_ERR on mlx5_0 qp 0x7fffe550f1c8.
> Mon Jan 16 14:19:02.051 2017: [E] VERBS RDMA closed connection to
> 10.101.53.20 (loremds20) on mlx5_0 port 1 fabnum 3 error 733 index 1
> Mon Jan 16 14:19:02.071 2017: [I] Recovered 2 nodes for file system tnb32.
> Mon Jan 16 14:19:02.140 2017: [I] VERBS RDMA connecting to 10.101.53.20
> (loremds20) on mlx5_0 port 1 fabnum 3 index 0
> Mon Jan 16 14:19:02.160 2017: [I] VERBS RDMA connected to 10.101.53.20
> (loremds20) on mlx5_0 port 1 fabnum 3 sl 0 index 0
>
> I had just shut down loremds18 and loremds19 so there was certainly
> recovery taking place and during that time is when the error seems to have
> occurred.
>
> I looked up the meaning of IBV_WC_REM_ACCESS_ERR here (
> http://www.rdmamojo.com/2013/02/15/ibv_poll_cq/) and see this:
>
> IBV_WC_REM_ACCESS_ERR (10) - Remote Access Error: a protection error
> occurred on a remote data buffer to be read by an RDMA Read, written by an
> RDMA Write or accessed by an atomic operation. This error is reported only
> on RDMA operations or atomic operations. Relevant for RC QPs.
>
> my take on it during recovery it seems like one end of the connection more
> or less hanging up on the other end (e.g. Connection reset by peer
> /ECONNRESET).
>
> But like I said at the start, we also see this when there something has
> gone awfully wrong.
>
> -Aaron
>
> On 1/18/17 3:59 AM, Simon Thompson (Research Computing - IT Services)
> wrote:
>
>> I'd be inclined to look at something like:
>>
>> ibqueryerrors -s
>> PortXmitWait,LinkDownedCounter,PortXmitDiscards,PortRcvRemot
>> ePhysicalErrors
>> -c
>>
>> And see if you have a high number of symbol errors, might be a cable
>> needs replugging or replacing.
>>
>> Simon
>>
>> From: > > on behalf of "J. Eric
>> Wonderley" >
>> Reply-To: "gpfsug-discuss@spectrumscale.org
>> "
>> > mscale.org>>
>> Date: Tuesday, 17 January 2017 at 21:16
>> To: "gpfsug-discuss@spectrumscale.org
>> "
>> > mscale.org>>
>> Subject: [gpfsug-discuss] rmda errors scatter thru gpfs logs
>>
>> I have messages like these frequent my logs:
>> Tue Jan 17 11:25:49.731 2017: [E] VERBS RDMA rdma write error
>> IBV_WC_REM_ACCESS_ERR to 10.51.10.5 (cl005) on mlx5_0 port 1 fabnum 0
>> vendor_err 136
>> Tue Jan 17 11:25:49.732 2017: [E] VERBS RDMA closed connection to
>> 10.51.10.5 (cl005) on mlx5_0 port 1 fabnum 0 due to RDMA write error
>> IBV_WC_REM_ACCESS_ERR index 23
>>
>> Any ideas on cause..?
>>
>>
>>
>> ___
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>>
> --
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Mounting GPFS data on OpenStack VM

[Brian Marshall]

Perfect.  Thanks for the advice.

Further: this might be a basic question: Are their design guides for
building CES protocl servers?

Brian

On Fri, Jan 20, 2017 at 1:04 PM, Gaurang Tapase <gaurang.tap...@in.ibm.com>
wrote:

> Hi Brian,
>
> For option #3, you can use GPFS Manila (OpenStack shared file system
> service) driver for exporting data from protocol servers to the OpenStack
> VMs.
> It was updated to support CES in the Newton release.
>
> A new feature of bringing existing filesets under Manila management has
> also been added recently.
>
> Thanks,
> Gaurang
> 
> Gaurang S Tapase
> Spectrum Scale & OpenStack
> IBM India Storage Lab, Pune (India)
> Email : gaurang.tap...@in.ibm.com
> Phone : +91-20-42025699 <+91%2020%204202%205699> (W), +91-9860082042
> <+91%2098600%2082042>(Cell)
> -----
>
>
>
> From:Brian Marshall <mimar...@vt.edu>
> To:gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>
> Date:01/18/2017 09:52 PM
> Subject:Re: [gpfsug-discuss] Mounting GPFS data on OpenStack VM
> Sent by:gpfsug-discuss-boun...@spectrumscale.org
> --
>
>
>
> To answer some more questions:
>
> What sort of workload will your Nova VM's be running?
> This is largely TBD but we anticipate webapps and other non-batch ways of
> interacting with and post processing data that has been computed on HPC
> batch systems.  For example a user might host a website that allows users
> to view pieces of a large data set and do some processing in private cloud
> or kick off larger jobs on HPC clusters
>
> How many VM's are you running?
> This work is still in the design / build phase.  We have 48 servers slated
> for the project.  At max maybe 500 VMs; again this is a pretty wild
> estimate.  This is a new service we are looking to provide
>
> What is your Network interconnect between the Scale Storage cluster and
> the Nova Compute cluster
> Each nova node has a dual 10gigE connection to switches that uplink to our
> core 40 gigE switches were NSD Servers are directly connectly.
>
> The information so far has been awesome.  Thanks everyone.  I am
> definitely leaning towards option #3 of creating protocol servers.  Are
> there any design/build white papers targetting the virutalization use case?
>
> Thanks,
> Brian
>
> On Tue, Jan 17, 2017 at 5:55 PM, Andrew Beattie <*abeat...@au1.ibm.com*
> <abeat...@au1.ibm.com>> wrote:
> HI Brian,
>
>
> Couple of questions for you:
>
> What sort of workload will your Nova VM's be running?
> How many VM's are you running?
> What is your Network interconnect between the Scale Storage cluster and
> the Nova Compute cluster
>
> I have cc'd Jake Carrol from University of Queensland in on the email as I
> know they have done some basic performance testing using Scale to provide
> storage to Openstack.
> One of the issues that they found was the Openstack network translation
> was a performance limiting factor.
>
> I think from memory the best performance scenario they had was, when they
> installed the scale client locally into the virtual machines
>
>
> *Andrew Beattie*
> *Software Defined Storage  - IT Specialist*
> *Phone: *614-2133-7927
> *E-mail: **abeat...@au1.ibm.com* <abeat...@au1.ibm.com>
>
>
> - Original message -
> From: Brian Marshall <*mimar...@vt.edu* <mimar...@vt.edu>>
> Sent by: *gpfsug-discuss-boun...@spectrumscale.org*
> <gpfsug-discuss-boun...@spectrumscale.org>
> To: gpfsug main discussion list <*gpfsug-discuss@spectrumscale.org*
> <gpfsug-discuss@spectrumscale.org>>
> Cc:
> Subject: [gpfsug-discuss] Mounting GPFS data on OpenStack VM
> Date: Wed, Jan 18, 2017 7:51 AM
>
> UG,
>
> I have a GPFS filesystem.
>
> I have a OpenStack private cloud.
>
> What is the best way for Nova Compute VMs to have access to data inside
> the GPFS filesystem?
>
> 1)Should VMs mount GPFS directly with a GPFS client?
> 2) Should the hypervisor mount GPFS and share to nova computes?
> 3) Should I create GPFS protocol servers that allow nova computes to mount
> of NFS?
>
> All advice is welcome.
>
>
> Best,
> Brian Marshall
> Virginia Tech
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at *spectrumscale.org* <http://spectrumscale.org/>
> *http://gpfsug.org/mailman/listinfo/gpfsug-discuss*
> <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>
>
> ___

[gpfsug-discuss] Data Replication

[Brian Marshall]

All,

If I setup a filesystem to have data replication of 2 (2 copies of data),
does the data get replicated at the NSD Server or at the client?  i.e. Does
the client send 2 copies over the network or does the NSD Server get a
single copy and then replicate on storage NSDs?

I couldn't find a place in the docs that talked about this specific point.

Thank you,
Brian Marshall
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Data Replication

[Brian Marshall]

Thanks.   This confirms the numbers that I am seeing.

Brian

On Tue, Aug 30, 2016 at 2:50 PM, Laurence Horrocks-Barlow <
laure...@qsplace.co.uk> wrote:

> Its the client that does all the synchronous replication, this way the
> cluster is able to scale as the clients do the leg work (so to speak).
>
> The somewhat "exception" is if a GPFS NSD server (or client with direct
> NSD) access uses a server bases protocol such as SMB, in this case the SMB
> server will do the replication as the SMB client doesn't know about GPFS or
> its replication; essentially the SMB server is the GPFS client.
>
> -- Lauz
>
> On 30 August 2016 17:03:38 CEST, Bryan Banister <bbanis...@jumptrading.com>
> wrote:
>
>> The NSD Client handles the replication and will, as you stated, write one
>> copy to one NSD (using the primary server for this NSD) and one to a
>> different NSD in a different GPFS failure group (using quite likely, but
>> not necessarily, a different NSD server that is the primary server for this
>> alternate NSD).
>>
>> Cheers,
>>
>> -Bryan
>>
>>
>>
>> *From:* gpfsug-discuss-boun...@spectrumscale.org [mailto:gpfsug-discuss-
>> boun...@spectrumscale.org] *On Behalf Of *Brian Marshall
>> *Sent:* Tuesday, August 30, 2016 9:59 AM
>> *To:* gpfsug main discussion list
>> *Subject:* [gpfsug-discuss] Data Replication
>>
>>
>>
>> All,
>>
>>
>>
>> If I setup a filesystem to have data replication of 2 (2 copies of data),
>> does the data get replicated at the NSD Server or at the client?  i.e. Does
>> the client send 2 copies over the network or does the NSD Server get a
>> single copy and then replicate on storage NSDs?
>>
>>
>>
>> I couldn't find a place in the docs that talked about this specific point.
>>
>>
>>
>> Thank you,
>>
>> Brian Marshall
>>
>>
>> --
>>
>> Note: This email is for the confidential use of the named addressee(s)
>> only and may contain proprietary, confidential or privileged information.
>> If you are not the intended recipient, you are hereby notified that any
>> review, dissemination or copying of this email is strictly prohibited, and
>> to please notify the sender immediately and destroy this email and any
>> attachments. Email transmission cannot be guaranteed to be secure or
>> error-free. The Company, therefore, does not make any guarantees as to the
>> completeness or accuracy of this email or any attachments. This email is
>> for informational purposes only and does not constitute a recommendation,
>> offer, request or solicitation of any kind to buy, sell, subscribe, redeem
>> or perform any type of transaction of a financial product.
>>
>> --
>>
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Data Replication

[Brian Marshall]

Daniel,

So here's my use case:  I have a Sandisk IF150 (branded as DeepFlash
recently) with 128TB of flash acting as a "fast tier" storage pool in our
HPC scratch file system.  Can I set the filesystem replication level to 1
then write a policy engine rule to send small and/or recent files to the
IF150 with a replication of 2?

Any other comments on the proposed usage strategy are helpful.

Thank you,
Brian Marshall

On Wed, Aug 31, 2016 at 10:32 AM, Daniel Kidger <daniel.kid...@uk.ibm.com>
wrote:

> The other 'Exception' is when a rule is used to convert a 1 way replicated
> file to 2 way, or when only one failure group is up due to HW problems. It
> that case the (re-replication) is done by whatever nodes are used for the
> rule or command-line, which may include an NSD server.
>
> Daniel
>
> IBM Spectrum Storage Software
> +44 (0)7818 522266 <+44%207818%20522266>
> Sent from my iPad using IBM Verse
>
>
> --
> On 30 Aug 2016, 19:53:31, mimar...@vt.edu wrote:
>
> From: mimar...@vt.edu
> To: gpfsug-discuss@spectrumscale.org
> Cc:
> Date: 30 Aug 2016 19:53:31
> Subject: Re: [gpfsug-discuss] Data Replication
>
>
> Thanks.   This confirms the numbers that I am seeing.
>
> Brian
>
> On Tue, Aug 30, 2016 at 2:50 PM, Laurence Horrocks-Barlow <
> laure...@qsplace.co.uk> wrote:
>
>> Its the client that does all the synchronous replication, this way the
>> cluster is able to scale as the clients do the leg work (so to speak).
>>
>> The somewhat "exception" is if a GPFS NSD server (or client with direct
>> NSD) access uses a server bases protocol such as SMB, in this case the SMB
>> server will do the replication as the SMB client doesn't know about GPFS or
>> its replication; essentially the SMB server is the GPFS client.
>>
>> -- Lauz
>>
>> On 30 August 2016 17:03:38 CEST, Bryan Banister <
>> bbanis...@jumptrading.com> wrote:
>>
>>> The NSD Client handles the replication and will, as you stated, write
>>> one copy to one NSD (using the primary server for this NSD) and one to a
>>> different NSD in a different GPFS failure group (using quite likely, but
>>> not necessarily, a different NSD server that is the primary server for this
>>> alternate NSD).
>>>
>>> Cheers,
>>>
>>> -Bryan
>>>
>>>
>>>
>>> *From:* gpfsug-discuss-boun...@spectrumscale.org [mailto:
>>> gpfsug-discuss-boun...@spectrumscale.org] *On Behalf Of *Brian Marshall
>>> *Sent:* Tuesday, August 30, 2016 9:59 AM
>>> *To:* gpfsug main discussion list
>>> *Subject:* [gpfsug-discuss] Data Replication
>>>
>>>
>>>
>>> All,
>>>
>>>
>>>
>>> If I setup a filesystem to have data replication of 2 (2 copies of
>>> data), does the data get replicated at the NSD Server or at the client?
>>>  i.e. Does the client send 2 copies over the network or does the NSD Server
>>> get a single copy and then replicate on storage NSDs?
>>>
>>>
>>>
>>> I couldn't find a place in the docs that talked about this specific
>>> point.
>>>
>>>
>>>
>>> Thank you,
>>>
>>> Brian Marshall
>>>
>>>
>>> --
>>>
>>> Note: This email is for the confidential use of the named addressee(s)
>>> only and may contain proprietary, confidential or privileged information.
>>> If you are not the intended recipient, you are hereby notified that any
>>> review, dissemination or copying of this email is strictly prohibited, and
>>> to please notify the sender immediately and destroy this email and any
>>> attachments. Email transmission cannot be guaranteed to be secure or
>>> error-free. The Company, therefore, does not make any guarantees as to the
>>> completeness or accuracy of this email or any attachments. This email is
>>> for informational purposes only and does not constitute a recommendation,
>>> offer, request or solicitation of any kind to buy, sell, subscribe, redeem
>>> or perform any type of transaction of a financial product.
>>>
>>> --
>>>
>>> gpfsug-discuss mailing list
>>> gpfsug-discuss at spectrumscale.org
>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>>
>>>
>> --
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>
>> ___
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

[gpfsug-discuss] EDR and omnipath

[Brian Marshall]

All,

I see in the GPFS FAQ A6.3 the statement below.  Is it possible to have
GPFS do RDMA over EDR infiniband and non-RDMA communication over omnipath
(IP over fabric) when each NSD server has an EDR card and a OPA card
installed?



RDMA is not supported on a node when both Mellanox HCAs and Intel Omni-Path
HFIs are enabled for RDMA.
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

[gpfsug-discuss] testing commands

[Brian Marshall]

All,

Is there a way to "test" GPFS commands and see what the output or result
would be before running them?  For example, I'd like to verify a command
string before actually running it on a production system.

Does IBM offer "test" licenses for setting up a small debug/devel
environment?

I'd be interested to hear everyone's best practices on verifying
commands/actions.


Thanks,
Brian
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

[gpfsug-discuss] NSD Server BIOS setting - snoop mode

[Brian Marshall]

All,

Is there any best practice or recommendation for the Snoop Mode memory
setting for NSD Servers?

Default is Early Snoop.   On compute nodes, I am using Cluster On Die,
which creates 2 NUMA nodes per processor.   This setup has 2 x 16-core
Broadwell processors in each NSD server.

Brian
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] GPFS FPO

[Brian Marshall]

Aaron,

Do you have experience running this on native GPFS?  The docs say Lustre
and any NFS filesystem.

Thanks,
Brian

On Aug 22, 2016 10:37 PM, "Aaron Knister" <aaron.s.knis...@nasa.gov> wrote:

> Yes, indeed. Note that these are my personal opinions.
>
> It seems to work quite well and it's not terribly hard to set up or get
> running. That said, if you've got a traditional HPC cluster with reasonably
> good bandwidth (and especially if your data is already on the HPC cluster)
> I wouldn't bother with FPO and just use something like magpie (
> https://github.com/LLNL/magpie) to run your hadoopy workload on GPFS on
> your traditional HPC cluster. I believe FPO (and by extension data
> locality) is important when the available bandwidth between your clients
> and servers/disks (in a traditional GPFS environment) is less than the
> bandwidth available within a node (e.g. between your local disks and the
> host CPU).
>
> -Aaron
>
> On 8/22/16 10:23 PM, Brian Marshall wrote:
>
>> Does anyone have any experiences to share (good or bad) about setting up
>> and utilizing FPO for hadoop compute on top of GPFS?
>>
>>
>> ___
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>>
> --
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

[gpfsug-discuss] subnets

[Brian Marshall]

All,

We are setting up communication between 2 clusters using ethernet and
IPoFabric.

The Daemon interface is running on ethernet, so all admin traffic will use
it.

We are still getting the subnets setting correct.

Question:

Does GPFS have a way to query how it is connecting to a given cluster/node?
 i.e. once we have subnets setup how can we tell GPFS is actually using
them.  Currently we just do a large transfer and check tcpdump for any
packets flowing on the high-speed/data/non-admin subnet.


Thank you,
Brian Marshall
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size

[Brian Marshall]

When creating a "fast tier" storage pool in a filesystem is the normal case
to create a placement policy that places all files in the fast tier and
migrates out old and large files?


Brian Marshall

On Mon, Oct 31, 2016 at 1:20 PM, Jez Tucker <jez.tuc...@gpfsug.org> wrote:

> Hey Bryan
>
>   There was a previous RFE for path placement from the UG, but Yuri told
> me this was not techically possible as an inode has no knowledge about the
> parent dentry.  (IIRC).You can see this in effect in the C API.  It is
> possible to work this out at kernel level, but it's so costly that it
> becomes non-viable at scale / performance.
>
> IBMers please chip in and expand if you will.
>
> Jez
>
>
> On 31/10/16 17:09, Bryan Banister wrote:
>
> The File Placement Policy that you are trying to set cannot use the size
> of the file to determine the placement of the file in a GPFS Storage Pool.
> This is because GPFS has no idea what the file size will be when the file
> is open()’d for writing.
>
>
>
> Hope that helps!
>
> -Bryan
>
>
>
> PS. I really wish that we could use a path for specifying data placement
> in a GPFS Pool, and not just the file name, owner, etc.  I’ll submit a RFE
> for this.
>
>
>
> *From:* gpfsug-discuss-boun...@spectrumscale.org [mailto:gpfsug-discuss-
> boun...@spectrumscale.org <gpfsug-discuss-boun...@spectrumscale.org>] *On
> Behalf Of *J. Eric Wonderley
> *Sent:* Monday, October 31, 2016 11:53 AM
> *To:* gpfsug main discussion list
> *Subject:* [gpfsug-discuss] wanted...gpfs policy that places larger files
> onto a pool based on size
>
>
>
> I wanted to do something like this...
>
>
> [root@cl001 ~]# cat /opt/gpfs/home.ply
> /*Failsafe migration of old small files back to spinning media pool(fc_8T)
> */
> RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70)
> WEIGHT(ACCESS_TIME) TO POOL 'fc_8T'
> /*Write files larger than 16MB to pool called "fc_8T" */
> RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216
> /*Move anything else to system pool */
> RULE 'default' SET POOL 'system'
>
> Apparently there is no happiness using FILE_SIZE in a placement policy:
> [root@cl001 ~]# mmchpolicy home /opt/gpfs/home.ply
> Error while validating policy `home.ply': rc=22:
> PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or variable
> name in this context.
> PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE
> {{{FILE_SIZE}}}>16777216
> runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home
> /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply   failed.
> mmchpolicy: Command failed. Examine previous error messages to determine
> cause.
>
> Can anyone suggest a way to accomplish this using policy?
>
> --
>
> Note: This email is for the confidential use of the named addressee(s)
> only and may contain proprietary, confidential or privileged information.
> If you are not the intended recipient, you are hereby notified that any
> review, dissemination or copying of this email is strictly prohibited, and
> to please notify the sender immediately and destroy this email and any
> attachments. Email transmission cannot be guaranteed to be secure or
> error-free. The Company, therefore, does not make any guarantees as to the
> completeness or accuracy of this email or any attachments. This email is
> for informational purposes only and does not constitute a recommendation,
> offer, request or solicitation of any kind to buy, sell, subscribe, redeem
> or perform any type of transaction of a financial product.
>
>
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at 
> spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] SC16 Draft Agenda [Save the date: Sunday, November 13th]

[Brian Marshall]

Virginia Tech ARC purchased a SANDisk IF150 (IBM DeepFlash 150) this Summer
and are just getting it into production (maybe next week).  I have some dev
benchmarking numbers and may have production/user feedback by SC16.  I'm
not sure I can fill an entire 30 minutes with worthwhile information, but I
can give at least 15 minutes on:

DeepFlash: a computational scientist's first impressions

Best,
Brian Marshall

On Mon, Oct 10, 2016 at 6:59 PM, GPFS UG USA Principal <
usa-princi...@gpfsug.org> wrote:

> Hello all,
>
> There have been some questions about the Spectrum Scale Users Group event
> for SC16 and we thought it best to publish our draft agenda and continue to
> fill it in as details get settled. That way you can make your travel plans
> and schedules.
>
> The date and rough times are set, we may adjust the time range a bit
> depending upon the number of user talks that people would like to give. So
> note, yes! *we are asking for user talks. Please consider doing this. *We
> always receive positive feedback about talks given by users that describe
> real world experiences. If *any* of the user talk slots below are of
> interest to you, please let us know as soon as you can. We may turn at
> least one user talk into a panel if we can get enough participants.
>
> As always, your feedback is welcome. This is a *users* group after all.
>
> So, here’s what we have planned so far:
>
> *Date: Sunday November 13th*
> *Venue: TBD, but near the convention center in downtown SLC*
> *Time: ~12:30p - ~17:30p*
>
> *Agenda:*
>
>
>
>
>
>
>
>
> *12:30 - 12:50 Welcome / Overview12:50 - 13:50 The latest in IBM Spectrum
> Scale and ESS13:50 - 14:20 User Talk: Big Data in HPC14:20 - 14:50 User
> Talk: Tiering to Object Storage14:50 - 15:20  - break -15:20 - 15:50
> User Talk:  TBD … volunteers! please! 15:50 - 16:35 Spectrum Scale
> Enhancements for CORAL16:35 - 17:20 News from IBM Research17:20 - 17:30
> Closing*
>
> Best,
>
> Kristy & Bob
>
> Kristy Kallback-Rose
> Manager, Research Storage
> Indiana University
>
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] searching for mmcp or mmcopy - optimized bulk copy for spectrum scale?

[Brian Marshall]

All,

I am in the same boat.  I'd like to copy ~500 TB from one filesystem to
another.  Both are being served by the same NSD servers.

We've done the multiple rsync script method in the past (and yes it's a bit
of a pain).  Would love to have an easier utility.

Best,
Brian Marshall

On Mon, Dec 5, 2016 at 5:26 AM, Heiner Billich <heiner.bill...@psi.ch>
wrote:

> Hello,
>
> I heard about some gpfs optimized bulk(?) copy command  named 'mmcp' or
> 'mmcopy' but couldn't find it in either /user/lpp/mmfs/samples/ or by
> asking google. Can please somebody point me to the source? I wonder whether
> it  allows incremental copies as rsync does.
>
> We need to copy a few 100TB of data and simple rsync provides just about
> 100MB/s. I know about the possible workarounds - write a wrapper script,
> run several rsyncs in parallel, distribute the rsync jobs on several nodes,
> use a special rsync versions that knows about gpfs ACLs, ... or try mmfind,
> which requires me to write a custom wrapper for cp 
>
> I really would prefer some ready-to-use script or program.
>
> Thank you and kind regards,
> Heiner Billich
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] replication and no failure groups

[Brian Marshall]

;
>> Yes   ready up
>>
>> system
>> nsd_b_11 nsd
>>
>> 512  -1 No
>>
>> Yes   ready up
>>
>> system
>> nsd_c_11 nsd
>>
>> 512  -1 No
>>
>> Yes   ready up
>>
>> system
>> nsd_d_11 nsd
>>
>> 512  -1 No
>>
>> Yes   ready up
>>
>> system
>> nsd_a_12 nsd
>>
>> 512  -1 No
>>
>> Yes   ready up
>>
>> system
>> nsd_b_12 nsd
>>
>> 512  -1 No
>>
>> Yes   ready up
>>
>> system
>> nsd_c_12 nsd
>>
>> 512  -1 No
>>
>> Yes   ready up
>>
>> system
>> nsd_d_12 nsd
>>
>> 512  -1 No
>>
>> Yes   ready up
>>
>> system
>> work_md_pf1_1 nsd 512
>>
>> 200 Yes  Noready
>>
>> up   system
>>
>>
>> jbf1z1   nsd
>>
>> 40962012 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf2z1   nsd
>>
>> 40962012 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf3z1   nsd
>>
>> 40962012 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf4z1   nsd
>>
>> 40962012 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf5z1   nsd
>>
>> 40962012 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf6z1   nsd
>>
>> 40962012 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf7z1   nsd
>>
>> 40962012 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf8z1   nsd
>>
>> 40962012 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf1z2   nsd
>>
>> 40962012 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf2z2   nsd
>>
>> 40962012 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf3z2   nsd
>>
>> 40962012 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf4z2   nsd
>>
>> 40962012 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf5z2   nsd
>>
>> 40962012 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf6z2   nsd
>>
>> 40962012 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf7z2   nsd
>>
>> 40962012 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf8z2   nsd
>>
>> 40962012 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf1z3   nsd
>>
>> 40962034 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf2z3   nsd
>>
>> 40962034 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf3z3   nsd
>>
>> 40962034 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf4z3   nsd
>>
>> 40962034 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf5z3   nsd
>>
>> 40962034 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf6z3   nsd
>>
>> 40962034 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf7z3   nsd
>>
>> 40962034 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf8z3   nsd
>>
>> 40962034 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf1z4   nsd
>>
>> 40962034 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf2z4   nsd
>>
>> 40962034 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf3z4   nsd
>>
>> 40962034 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf4z4   nsd
>>
>> 40962034 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf5z4   nsd
>>
>> 40962034 No
>>
>> Yes   ready up
>>
>> sas_ssd4T
>> jbf6z4   nsd
>>
>>

[gpfsug-discuss] replication and no failure groups

[Brian Marshall]

All,

If I have a filesystem with replication set to 2 and 1 failure group:

1) I assume replication won't actually happen, correct?

2) Will this impact performance i.e cut write performance in half even
though it really only keeps 1 copy?

End goal - I would like a single storage pool within the filesystem to be
replicated without affecting the performance of all other pools(which only
have a single failure group)

Thanks,
Brian Marshall
VT - ARC
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Tiers

[Brian Marshall]

We are in very similar situation.  VT - ARC has a layer of SSD for metadata
only,  another layer of SSD for "hot" data, and a layer of 8TB HDDs for
capacity.   We just now in the process of getting it all into production.

On this topic:

What is everyone's favorite migration policy to move data from SSD to HDD
(and vice versa)?

Do you nightly move large/old files to HDD or wait until the fast tier hit
some capacity limit?

Do you use QOS to limit the migration from SSD to HDD i.e. try not to kill
the file system with migration work?


Thanks,
Brian Marshall

On Thu, Dec 15, 2016 at 4:25 PM, Buterbaugh, Kevin L <
kevin.buterba...@vanderbilt.edu> wrote:

> Hi Mark,
>
> We just use an 8 Gb FC SAN.  For the data pool we typically have a dual
> active-active controller storage array fronting two big RAID 6 LUNs and 1
> RAID 1 (for /home).  For the capacity pool, it might be the same exact
> model of controller, but the two controllers are now fronting that whole
> 60-bay array.
>
> But our users tend to have more modest performance needs than most…
>
> Kevin
>
> On Dec 15, 2016, at 3:19 PM, mark.b...@siriuscom.com wrote:
>
> Kevin, out of curiosity, what type of disk does your data pool use?  SAS
> or just some SAN attached system?
>
> *From: *<gpfsug-discuss-boun...@spectrumscale.org> on behalf of
> "Buterbaugh, Kevin L" <kevin.buterba...@vanderbilt.edu>
> *Reply-To: *gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>
> *Date: *Thursday, December 15, 2016 at 2:47 PM
> *To: *gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>
> *Subject: *Re: [gpfsug-discuss] Tiers
>
> Hi Mark,
>
> We’re a “traditional” university HPC center with a very untraditional
> policy on our scratch filesystem … we don’t purge it and we sell quota
> there.  Ultimately, a lot of that disk space is taken up by stuff that,
> let’s just say, isn’t exactly in active use.
>
> So what we’ve done, for example, is buy a 60-bay storage array and stuff
> it with 8 TB drives.  It wouldn’t offer good enough performance for
> actively used files, but we use GPFS policies to migrate files to the
> “capacity” pool based on file atime.  So we have 3 pools:
>
> 1.  the system pool with metadata only (on SSDs)
> 2.  the data pool, which is where actively used files are stored and which
> offers decent performance
> 3.  the capacity pool, for data which hasn’t been accessed “recently”, and
> which is on slower storage
>
> I would imagine others do similar things.  HTHAL…
>
> Kevin
>
>
> On Dec 15, 2016, at 2:32 PM, mark.b...@siriuscom.com wrote:
>
> Just curious how many of you out there deploy SS with various tiers?  It
> seems like a lot are doing the system pool with SSD’s but do you routinely
> have clusters that have more than system pool and one more tier?
>
> I know if you are doing Archive in connection that’s an obvious choice for
> another tier but I’m struggling with knowing why someone needs more than
> two tiers really.
>
> I’ve read all the fine manuals as to how to do such a thing and some of
> the marketing as to maybe why.  I’m still scratching my head on this
> though.  In fact, my understanding is in the ESS there isn’t any different
> pools (tiers) as it’s all NL-SAS or SSD (DF150, etc).
>
> It does make sense to me know with TCT and I could create an ILM policy to
> get some of my data into the cloud.
>
> But in the real world I would like to know what yall do in this regard.
>
>
> Thanks
>
> Mark
>
> This message (including any attachments) is intended only for the use of
> the individual or entity to which it is addressed and may contain
> information that is non-public, proprietary, privileged, confidential, and
> exempt from disclosure under applicable law. If you are not the intended
> recipient, you are hereby notified that any use, dissemination,
> distribution, or copying of this communication is strictly prohibited. This
> message may be viewed by parties at Sirius Computer Solutions other than
> those named in the message header. This message does not contain an
> official representation of Sirius Computer Solutions. If you have received
> this communication in error, notify Sirius Computer Solutions immediately
> and (i) destroy this message if a facsimile or (ii) delete this message
> immediately if this is an electronic communication. Thank you.
> *Sirius Computer Solutions <http://www.siriuscom.com/>*
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> ___
> gpfsug-discuss mailing list
> gpfsug-discu

[gpfsug-discuss] mmlsdisk performance impact

[Brian Marshall]

All,

Does the mmlsdisk command generate a lot of admin traffic or take up a lot
of GPFS resources?

In our case, we have it in some of our monitoring routines that run on all
nodes.  It is kind of nice info to have, but I am wondering if hitting the
filesystem with a bunch of mmlsdisk commands is bad for performance.


Thanks,
Brian
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

[gpfsug-discuss] reserving memory for GPFS process

[Brian Marshall]

All,

What is your favorite method for stopping a user process from eating up all
the system memory and saving 1 GB (or more) for the GPFS / system
processes?  We have always kicked around the idea of cgroups but never
moved on it.

The problem:  A user launches a job which uses all the memory on a node,
which causes the node to be expelled, which causes brief filesystem
slowness everywhere.

I bet this problem has already been solved and I am just googling the wrong
search terms.


Thanks,
Brian
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss