The Spectrum Scale FAQ lists what is and is not supported, in Table 7:
https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html
LROC is available on all Editions; in the notes to Table 7 is the clarification:
"1. Functions which are supported by all Editions on all platforms are not listed in the matrix."
> by looking at the documentation it is not clear whether Spectrum Scale
> Express edition supports the LROC feature. As far as I understand it,
> the client license is sufficient, however no word is given about which
> edition supports that feature.
> Any idea and/or pointer?
> Many thanks,
> Pascal
Keith D. Ball, PhD
Client Technical Specialist, US Federal
IBM Systems, Software Defined Storage Solutions (SDSS)
Spectrum Scale (GPFS) - Spectrum Storage Software - Cleversafe - Elastic Storage Server
Platform Computing - High-Performance Cloud Services
IBM Certified Deployment Professional - Spectrum Scale V4.1.1
540-557-7851 cell
[email protected]
Client Technical Specialist, US Federal
IBM Systems, Software Defined Storage Solutions (SDSS)
Spectrum Scale (GPFS) - Spectrum Storage Software - Cleversafe - Elastic Storage Server
Platform Computing - High-Performance Cloud Services
IBM Certified Deployment Professional - Spectrum Scale V4.1.1
540-557-7851 cell
[email protected]
[email protected] wrote: -----
To: [email protected]
From: [email protected]
Sent by: [email protected]
Date: 11/09/2016 01:05PM
Subject: gpfsug-discuss Digest, Vol 58, Issue 12
From: [email protected]
Sent by: [email protected]
Date: 11/09/2016 01:05PM
Subject: gpfsug-discuss Digest, Vol 58, Issue 12
Send gpfsug-discuss mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."
Today's Topics:
1. LROC and Spectrum Scale Express (Pascal Jermini)
2. Tuning AFM for high throughput/high IO over _really_ long
distances (Jake Carroll)
3. Re: Tuning AFM for high throughput/high IO over _really_ long
distances (Jan-Frode Myklebust)
----------------------------------------------------------------------
------------------------------
Message: 2
Date: Wed, 9 Nov 2016 17:39:05 +0000
From: Jake Carroll <[email protected]>
To: "[email protected]"
<[email protected]>
Subject: [gpfsug-discuss] Tuning AFM for high throughput/high IO over
_really_ long distances
Message-ID: <[email protected]>
Content-Type: text/plain; charset="utf-8"
Hi.
I?ve got an GPFS to GPFS AFM cache/home (IW) relationship set up over a really long distance. About 180ms of latency between the two clusters and around 13,000km of optical path. Fortunately for me, I?ve actually got near theoretical maximum IO over the NIC?s between the clusters and I?m iPerf?ing at around 8.90 to 9.2Gbit/sec over a 10GbE circuit. All MTU9000 all the way through.
Anyway ? I?m finding my AFM traffic to be dragging its feet and I don?t really understand why that might be. I?ve verified the links and transports ability as I said above with iPerf, and CERN?s FDT to near 10Gbit/sec.
I also verified the clusters on both sides in terms of disk IO and they both seem easily capable in IOZone and IOR tests of multiple GB/sec of throughput.
So ? my questions:
1. Are there very specific tunings AFM needs for high latency/long distance IO?
2. Are there very specific NIC/TCP-stack tunings (beyond the type of thing we already have in place) that benefits AFM over really long distances and high latency?
3. We are seeing on the ?cache? side really lazy/sticky ?ls ?als? in the home mount. It sometimes takes 20 to 30 seconds before the command line will report back with a long listing of files. Any ideas why it?d take that long to get a response from ?home?.
We?ve got our TCP stack setup fairly aggressively, on all hosts that participate in these two clusters.
ethtool -C enp2s0f0 adaptive-rx off
ifconfig enp2s0f0 txqueuelen 10000
sysctl -w net.core.rmem_max=536870912
sysctl -w net.core.wmem_max=536870912
sysctl -w net.ipv4.tcp_rmem="4096 87380 268435456"
sysctl -w net.ipv4.tcp_wmem="4096 65536 268435456"
sysctl -w net.core.netdev_max_backlog=250000
sysctl -w net.ipv4.tcp_congestion_control=htcp
sysctl -w net.ipv4.tcp_mtu_probing=1
I modified a couple of small things on the AFM ?cache? side to see if it?d make a difference such as:
mmchconfig afmNumWriteThreads=4
mmchconfig afmNumReadThreads=4
But no difference so far.
Thoughts would be appreciated. I?ve done this before over much shorter distances (30Km) and I?ve flattened a 10GbE wire without really tuning?anything. Are my large in-flight-packets numbers/long-time-to-acknowledgement semantics going to hurt here? I really thought AFM might be well designed for exactly this kind of work at long distance *and* high throughput ? so I must be missing something!
-jc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161109/d4f4d9a7/attachment-0001.html>
------------------------------
Message: 3
Date: Wed, 09 Nov 2016 18:05:21 +0000
From: Jan-Frode Myklebust <[email protected]>
To: "[email protected]"
<[email protected]>
Subject: Re: [gpfsug-discuss] Tuning AFM for high throughput/high IO
over _really_ long distances
Message-ID:
<CAHwPathy=4z=jDXN5qa3ys+Z-_7n=tsjh7cz3zkzfwqmg34...@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Mostly curious, don't have experience in such environments, but ... Is this
AFM over NFS or NSD protocol? Might be interesting to try the other option
-- and also check how nsdperf performs over such distance/latency.
-jf
ons. 9. nov. 2016 kl. 18.39 skrev Jake Carroll <[email protected]>:
> Hi.
>
>
>
> I?ve got an GPFS to GPFS AFM cache/home (IW) relationship set up over a
> really long distance. About 180ms of latency between the two clusters and
> around 13,000km of optical path. Fortunately for me, I?ve actually got near
> theoretical maximum IO over the NIC?s between the clusters and I?m
> iPerf?ing at around 8.90 to 9.2Gbit/sec over a 10GbE circuit. All MTU9000
> all the way through.
>
>
>
> Anyway ? I?m finding my AFM traffic to be dragging its feet and I don?t
> really understand why that might be. I?ve verified the links and transports
> ability as I said above with iPerf, and CERN?s FDT to near 10Gbit/sec.
>
>
>
> I also verified the clusters on both sides in terms of disk IO and they
> both seem easily capable in IOZone and IOR tests of multiple GB/sec of
> throughput.
>
>
>
> So ? my questions:
>
>
>
> 1. Are there very specific tunings AFM needs for high latency/long
> distance IO?
>
> 2. Are there very specific NIC/TCP-stack tunings (beyond the type
> of thing we already have in place) that benefits AFM over really long
> distances and high latency?
>
> 3. We are seeing on the ?cache? side really lazy/sticky ?ls ?als?
> in the home mount. It sometimes takes 20 to 30 seconds before the command
> line will report back with a long listing of files. Any ideas why it?d take
> that long to get a response from ?home?.
>
>
>
> We?ve got our TCP stack setup fairly aggressively, on all hosts that
> participate in these two clusters.
>
>
>
> ethtool -C enp2s0f0 adaptive-rx off
>
> ifconfig enp2s0f0 txqueuelen 10000
>
> sysctl -w net.core.rmem_max=536870912
>
> sysctl -w net.core.wmem_max=536870912
>
> sysctl -w net.ipv4.tcp_rmem="4096 87380 268435456"
>
> sysctl -w net.ipv4.tcp_wmem="4096 65536 268435456"
>
> sysctl -w net.core.netdev_max_backlog=250000
>
> sysctl -w net.ipv4.tcp_congestion_control=htcp
>
> sysctl -w net.ipv4.tcp_mtu_probing=1
>
>
>
> I modified a couple of small things on the AFM ?cache? side to see if it?d
> make a difference such as:
>
>
>
> mmchconfig afmNumWriteThreads=4
>
> mmchconfig afmNumReadThreads=4
>
>
>
> But no difference so far.
>
>
>
> Thoughts would be appreciated. I?ve done this before over much shorter
> distances (30Km) and I?ve flattened a 10GbE wire without really
> tuning?anything. Are my large in-flight-packets
> numbers/long-time-to-acknowledgement semantics going to hurt here? I really
> thought AFM might be well designed for exactly this kind of work at long
> distance **and** high throughput ? so I must be missing something!
>
>
>
> -jc
>
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161109/f44369ab/attachment.html>
------------------------------
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
End of gpfsug-discuss Digest, Vol 58, Issue 12
**********************************************
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."
Today's Topics:
1. LROC and Spectrum Scale Express (Pascal Jermini)
2. Tuning AFM for high throughput/high IO over _really_ long
distances (Jake Carroll)
3. Re: Tuning AFM for high throughput/high IO over _really_ long
distances (Jan-Frode Myklebust)
----------------------------------------------------------------------
------------------------------
Message: 2
Date: Wed, 9 Nov 2016 17:39:05 +0000
From: Jake Carroll <[email protected]>
To: "[email protected]"
<[email protected]>
Subject: [gpfsug-discuss] Tuning AFM for high throughput/high IO over
_really_ long distances
Message-ID: <[email protected]>
Content-Type: text/plain; charset="utf-8"
Hi.
I?ve got an GPFS to GPFS AFM cache/home (IW) relationship set up over a really long distance. About 180ms of latency between the two clusters and around 13,000km of optical path. Fortunately for me, I?ve actually got near theoretical maximum IO over the NIC?s between the clusters and I?m iPerf?ing at around 8.90 to 9.2Gbit/sec over a 10GbE circuit. All MTU9000 all the way through.
Anyway ? I?m finding my AFM traffic to be dragging its feet and I don?t really understand why that might be. I?ve verified the links and transports ability as I said above with iPerf, and CERN?s FDT to near 10Gbit/sec.
I also verified the clusters on both sides in terms of disk IO and they both seem easily capable in IOZone and IOR tests of multiple GB/sec of throughput.
So ? my questions:
1. Are there very specific tunings AFM needs for high latency/long distance IO?
2. Are there very specific NIC/TCP-stack tunings (beyond the type of thing we already have in place) that benefits AFM over really long distances and high latency?
3. We are seeing on the ?cache? side really lazy/sticky ?ls ?als? in the home mount. It sometimes takes 20 to 30 seconds before the command line will report back with a long listing of files. Any ideas why it?d take that long to get a response from ?home?.
We?ve got our TCP stack setup fairly aggressively, on all hosts that participate in these two clusters.
ethtool -C enp2s0f0 adaptive-rx off
ifconfig enp2s0f0 txqueuelen 10000
sysctl -w net.core.rmem_max=536870912
sysctl -w net.core.wmem_max=536870912
sysctl -w net.ipv4.tcp_rmem="4096 87380 268435456"
sysctl -w net.ipv4.tcp_wmem="4096 65536 268435456"
sysctl -w net.core.netdev_max_backlog=250000
sysctl -w net.ipv4.tcp_congestion_control=htcp
sysctl -w net.ipv4.tcp_mtu_probing=1
I modified a couple of small things on the AFM ?cache? side to see if it?d make a difference such as:
mmchconfig afmNumWriteThreads=4
mmchconfig afmNumReadThreads=4
But no difference so far.
Thoughts would be appreciated. I?ve done this before over much shorter distances (30Km) and I?ve flattened a 10GbE wire without really tuning?anything. Are my large in-flight-packets numbers/long-time-to-acknowledgement semantics going to hurt here? I really thought AFM might be well designed for exactly this kind of work at long distance *and* high throughput ? so I must be missing something!
-jc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161109/d4f4d9a7/attachment-0001.html>
------------------------------
Message: 3
Date: Wed, 09 Nov 2016 18:05:21 +0000
From: Jan-Frode Myklebust <[email protected]>
To: "[email protected]"
<[email protected]>
Subject: Re: [gpfsug-discuss] Tuning AFM for high throughput/high IO
over _really_ long distances
Message-ID:
<CAHwPathy=4z=jDXN5qa3ys+Z-_7n=tsjh7cz3zkzfwqmg34...@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Mostly curious, don't have experience in such environments, but ... Is this
AFM over NFS or NSD protocol? Might be interesting to try the other option
-- and also check how nsdperf performs over such distance/latency.
-jf
ons. 9. nov. 2016 kl. 18.39 skrev Jake Carroll <[email protected]>:
> Hi.
>
>
>
> I?ve got an GPFS to GPFS AFM cache/home (IW) relationship set up over a
> really long distance. About 180ms of latency between the two clusters and
> around 13,000km of optical path. Fortunately for me, I?ve actually got near
> theoretical maximum IO over the NIC?s between the clusters and I?m
> iPerf?ing at around 8.90 to 9.2Gbit/sec over a 10GbE circuit. All MTU9000
> all the way through.
>
>
>
> Anyway ? I?m finding my AFM traffic to be dragging its feet and I don?t
> really understand why that might be. I?ve verified the links and transports
> ability as I said above with iPerf, and CERN?s FDT to near 10Gbit/sec.
>
>
>
> I also verified the clusters on both sides in terms of disk IO and they
> both seem easily capable in IOZone and IOR tests of multiple GB/sec of
> throughput.
>
>
>
> So ? my questions:
>
>
>
> 1. Are there very specific tunings AFM needs for high latency/long
> distance IO?
>
> 2. Are there very specific NIC/TCP-stack tunings (beyond the type
> of thing we already have in place) that benefits AFM over really long
> distances and high latency?
>
> 3. We are seeing on the ?cache? side really lazy/sticky ?ls ?als?
> in the home mount. It sometimes takes 20 to 30 seconds before the command
> line will report back with a long listing of files. Any ideas why it?d take
> that long to get a response from ?home?.
>
>
>
> We?ve got our TCP stack setup fairly aggressively, on all hosts that
> participate in these two clusters.
>
>
>
> ethtool -C enp2s0f0 adaptive-rx off
>
> ifconfig enp2s0f0 txqueuelen 10000
>
> sysctl -w net.core.rmem_max=536870912
>
> sysctl -w net.core.wmem_max=536870912
>
> sysctl -w net.ipv4.tcp_rmem="4096 87380 268435456"
>
> sysctl -w net.ipv4.tcp_wmem="4096 65536 268435456"
>
> sysctl -w net.core.netdev_max_backlog=250000
>
> sysctl -w net.ipv4.tcp_congestion_control=htcp
>
> sysctl -w net.ipv4.tcp_mtu_probing=1
>
>
>
> I modified a couple of small things on the AFM ?cache? side to see if it?d
> make a difference such as:
>
>
>
> mmchconfig afmNumWriteThreads=4
>
> mmchconfig afmNumReadThreads=4
>
>
>
> But no difference so far.
>
>
>
> Thoughts would be appreciated. I?ve done this before over much shorter
> distances (30Km) and I?ve flattened a 10GbE wire without really
> tuning?anything. Are my large in-flight-packets
> numbers/long-time-to-acknowledgement semantics going to hurt here? I really
> thought AFM might be well designed for exactly this kind of work at long
> distance **and** high throughput ? so I must be missing something!
>
>
>
> -jc
>
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161109/f44369ab/attachment.html>
------------------------------
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
End of gpfsug-discuss Digest, Vol 58, Issue 12
**********************************************
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
