Thanks, Sven. Understood! On 12/19/17 3:20 PM, Sven Oehme wrote: > Hi, > > the zero padding was never promoted into a GA stream, it was an > experiment to proof we are on the right track when we eliminate the > overhead from client to NSD Server, but also showed that alone is not > good enough. the work for the client is the same compared to the >32 > subblocks, but the NSD Server has more work as it can't pack as many > subblocks and therefore files into larger blocks, so you need to do more > writes to store the same number of files. > thats why there is the additional substantial improvement when we then > went to >32 subblocks. > > sven > > On Mon, Dec 18, 2017 at 9:13 PM Knister, Aaron S. (GSFC-606.2)[COMPUTER > SCIENCE CORP] <[email protected] > <mailto:[email protected]>> wrote: > > Thanks Sven! That makes sense to me and is what I thought was the > case which is why I was confused when I saw the reply to the thread > that said the >32 subblocks code had no performance impact. > > A couple more question for you— in your presentation there’s a > benchmark that shows the file create performance without the zero > padding. Since you mention this is done for security reasons was > that feature ever promoted to a GA Scale release? I’m also wondering > if you could explain the performance difference between the no zero > padding code and the > 32 subblock code since given your the example > of 32K files and 16MB block size I figure both cases ought to write > the same amount to disk. > > Thanks! > > -Aaron > > > > > > On December 15, 2017 at 18:07:23 EST, Sven Oehme <[email protected] > <mailto:[email protected]>> wrote: >> i thought i answered that already, but maybe i just thought about >> answering it and then forgot about it :-D >> >> so yes more than 32 subblocks per block significant increase the >> performance of filesystems with small files, for the sake of the >> argument let's say 32k in a large block filesystem again for sake >> of argument say 16MB. >> >> you probably ask why ? >> >> if you create a file and write 32k into it in a pre 5.0.0 Version >> 16 MB filesystem your client actually doesn't write 32k to the NSD >> Server, it writes 512k, because thats the subblock size and we >> need to write the full subblock (for security reasons). so first >> you waste significant memory on the client to cache that zero >> padding, you waste network bandwidth and you waste NSD Server >> cache because you store it there too. this means you overrun the >> cache more quickly, means you start doing read/modify writes >> earlier on all your nice large raid tracks... i guess you get the >> story by now. >> >> in fact, if you have a good raid code that can drive really a lot >> of bandwidth out of individual drives like a GNR system you get >> more performance for small file writes as larger your blocksize >> is, because we can 'pack' more files into larger i/os and >> therefore turn a small file create workload into a bandwidth >> workload, essentially exactly what we did and i demonstrated in >> the CORAL presentation . >> >> hope that makes this crystal clear now . >> >> sven >> >> >> >> On Fri, Dec 15, 2017 at 10:47 PM Aaron Knister >> <[email protected] <mailto:[email protected]>> wrote: >> >> Thanks, Alex. I'm all too familiar with the trade offs between >> large >> blocks and small files and we do use pretty robust SSD storage >> for our >> metadata. We support a wide range of workloads and we have >> some folks >> with many small (<1M) files and other folks with many large >> (>256MB) files. >> >> My point in this thread is that IBM has said over and over >> again in >> presentations that there is a significant performance gain >> with the >32 >> subblocks code on filesystems with large block sizes (although >> to your >> point I'm not clear on exactly what large means since I didn't >> define >> large in this context). Therefore given that the >32 subblock >> code gives >> a significant performance gain one could reasonably assume >> that having a >> filesystem with >32 subblocks is required to see this gain >> (rather than >> just running the >32 subblocks code on an fs w/o > 32 subblocks). >> >> This lead me to ask about a migration tool because in my mind >> if there's >> a performance gain from having >32 subblocks on the FS I'd >> like that >> feature and having to manually copy 10's of PB to new hardware >> to get >> this performance boost is unacceptable. However, IBM can't >> seem to make >> up their mind about whether or not the >32 subblocks code >> *actually* >> provides a performance increase. This seems like a pretty >> straightforward question. >> >> -Aaron >> >> On 12/15/17 3:48 PM, Alex Chekholko wrote: >> > Hey Aaron, >> > >> > Can you define your sizes for "large blocks" and "small >> files"? If you >> > dial one up and the other down, your performance will be >> worse. And in >> > any case it's a pathological corner case so it shouldn't >> matter much for >> > your workflow, unless you've designed your system with the >> wrong values. >> > >> > For example, for bioinformatics workloads, I prefer to use 256KB >> > filesystem block size, and I'd consider 4MB+ to be "large >> block size", >> > which would make the filesystem obviously unsuitable for >> processing >> > millions of 8KB files. >> > >> > You can make a histogram of file sizes in your existing >> filesystems and >> > then make your subblock size (1/32 of block size) on the >> smaller end of >> > that. Also definitely use the "small file in inode" >> feature and put >> > your metadata on SSD. >> > >> > Regards, >> > Alex >> > >> > On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister >> > <[email protected] <mailto:[email protected]> >> <mailto:[email protected] >> <mailto:[email protected]>>> wrote: >> > >> > Thanks, Bill. >> > >> > I still don't feel like I've got an clear answer from >> IBM and frankly >> > the core issue of a lack of migration tool was totally >> dodged. >> > >> > Again in Sven's presentation from SSUG @ SC17 >> > >> >> (http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf >> >> <http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf>) >> > he mentions "It has a significant performance penalty >> for small files in >> > large block size filesystems" and the demonstrates that >> with several >> > mdtest runs (which show the effect with and without the >32 >> > subblocks code): >> > >> > >> > 4.2.1 base code - SUMMARY: (of 3 iterations) >> > File creation : Mean = 2237.644 >> > >> > zero-end-of-file-padding (4.2.2 + ifdef for zero >> padding): SUMMARY: (of >> > 3 iterations) >> > File creation : Mean = 12866.842 >> > >> > more sub blocks per block (4.2.2 + morethan32subblock code): >> > File creation : Mean = 40316.721 >> > >> > Can someone (ideally Sven) give me a straight answer as >> to whether or >> > not the > 32 subblock code actually makes a performance >> difference for >> > small files in large block filesystems? And if not, help >> me understand >> > why his slides and provided benchmark data have >> consistently indicated >> > it does? >> > >> > -Aaron >> > >> > On 12/1/17 11:44 AM, Bill Hartner wrote: >> > > ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported >> some of the >> > > results @ user group meeting. >> > > >> > > -Bill >> > > >> > > Bill Hartner >> > > IBM Systems >> > > Scalable I/O Development >> > > Austin, Texas >> > > [email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>> >> > > home office 512-784-0980 <tel:(512)%20784-0980> >> <tel:512-784-0980 <tel:(512)%20784-0980>> >> > > >> > > >> > > Inactive hide details for Jan-Frode Myklebust >> ---12/01/2017 06:53:44 >> > > AM---Bill, could you say something about what the >> metadataJan-Frode >> > > Myklebust ---12/01/2017 06:53:44 AM---Bill, could you >> say something >> > > about what the metadata-storage here was? >> ESS/NL-SAS/3way replication? >> > > >> > > From: Jan-Frode Myklebust <[email protected] >> <mailto:[email protected]> <mailto:[email protected] >> <mailto:[email protected]>>> >> > > To: gpfsug main discussion list >> <[email protected] >> <mailto:[email protected]> >> > <mailto:[email protected] >> <mailto:[email protected]>>> >> > > Date: 12/01/2017 06:53 AM >> > > Subject: Re: [gpfsug-discuss] Online data migration tool >> > > Sent by: [email protected] >> <mailto:[email protected]> >> > <mailto:[email protected] >> <mailto:[email protected]>> >> > > >> > > >> > >> >> ------------------------------------------------------------------------ >> > > >> > > >> > > >> > > Bill, could you say something about what the >> metadata-storage here was? >> > > ESS/NL-SAS/3way replication? >> > > >> > > I just asked about this in the internal slack channel >> #scale-help today.. >> > > >> > > >> > > >> > > -jf >> > > >> > > fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner >> <[email protected]_ >> > > <mailto:[email protected] >> <mailto:[email protected]> <mailto:[email protected] >> <mailto:[email protected]>>>>: >> > > >> > > > "It has a significant performance penalty for >> small files in >> > large >> > > > block size filesystems" >> > > >> > > Aaron, >> > > >> > > Below are mdtest results for a test we ran for >> CORAL - file >> > size was >> > > 32k. >> > > >> > > We have not gone back and ran the test on a file >> system formatted >> > > without > 32 subblocks. We'll do that at some point... >> > > >> > > -Bill >> > > >> > > -- started at 10/28/2017 17:51:38 -- >> > > >> > > mdtest-1.9.3 was launched with 228 total task(s) >> on 12 node(s) >> > > Command line used: /tmp/mdtest-binary-dir/mdtest -d >> > > /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w >> 32768 -C -F -r >> > -p 360 >> > > -u -y >> > > Path: /ibm/fs2-16m-10 >> > > FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used >> Inodes: 0.0% >> > > >> > > 228 tasks, 67239936 files >> > > >> > > SUMMARY: (of 3 iterations) >> > > Operation Max Min Mean Std Dev >> > > --------- --- --- ---- ------- >> > > File creation : 51953.498 50558.517 51423.221 616.643 >> > > File stat : 0.000 0.000 0.000 0.000 >> > > File read : 0.000 0.000 0.000 0.000 >> > > File removal : 96746.376 92149.535 94658.774 1900.187 >> > > Tree creation : 1.588 0.070 0.599 0.700 >> > > Tree removal : 0.213 0.034 0.097 0.082 >> > > >> > > -- finished at 10/28/2017 19:51:54 -- >> > > >> > > Bill Hartner >> > > IBM Systems >> > > Scalable I/O Development >> > > Austin, Texas_ >> > > [email protected]_ <mailto:[email protected] >> <mailto:[email protected]> >> > <mailto:[email protected] <mailto:[email protected]>>> >> > > home office 512-784-0980 <tel:(512)%20784-0980> >> <tel:512-784-0980 <tel:(512)%20784-0980>> >> > > >> > > _ >> > > [email protected]_ >> > > <mailto:[email protected] >> <mailto:[email protected]> >> > <mailto:[email protected] >> <mailto:[email protected]>>> wrote on >> > > 11/29/2017 04:41:48 PM: >> > > >> > > > From: Aaron Knister <[email protected]_ >> > > <mailto:[email protected] >> <mailto:[email protected]> >> <mailto:[email protected] >> <mailto:[email protected]>>>> >> > > >> > > >> > > > To: gpfsug main discussion list >> > > <[email protected]_ >> > > <mailto:[email protected] >> <mailto:[email protected]> >> > <mailto:[email protected] >> <mailto:[email protected]>>>> >> > > >> > > > Date: 11/29/2017 04:42 PM >> > > >> > > >> > > > Subject: Re: [gpfsug-discuss] Online data >> migration tool >> > > > Sent by: [email protected]_ >> > > <mailto:[email protected] >> <mailto:[email protected]> >> > <mailto:[email protected] >> <mailto:[email protected]>>> >> > > >> > > > >> > > >> > > > Thanks, Nikhil. Most of that was consistent with >> my understnading, >> > > > however I was under the impression that the >32 >> subblocks code is >> > > > required to achieve the touted 50k file >> creates/second that Sven has >> > > > talked about a bunch of times: >> > > > >> > > > >> > > >> > >> >> _http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_ >> > >> >> <http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_> >> > > >> > >> >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e= >> > >> >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=>> >> > > > >> > > >> > >> >> _http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_ >> > >> >> <http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_> >> > >> >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e= >> > >> >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=>> >> > > > _Sven_Oehme_-_News_from_Research.pdf >> > > > >> _http://files.gpfsug.org/presentations/2016/SC16/12_-_ >> > <http://files.gpfsug.org/presentations/2016/SC16/12_-_> >> > > >> > >> >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e= >> > >> >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=>> >> > > > >> _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf >> > > >> > > >> > > > from those presentations regarding 32 subblocks: >> > > > >> > > > "It has a significant performance penalty for >> small files in large >> > > > block size filesystems" >> > > >> > > > although I'm not clear on the specific >> definition of "large". Many >> > > > filesystems I encounter only have a 1M block >> size so it may not >> > > > matter there, although that same presentation >> clearly shows the >> > > > benefit of larger block sizes which is yet >> *another* thing for which >> > > > a migration tool would be helpful. >> > > >> > > > -Aaron >> > > > >> > > > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal >> > > <[email protected]_ <mailto:[email protected] >> <mailto:[email protected]> >> > <mailto:[email protected] >> <mailto:[email protected]>>>> wrote: >> > > >> > > > Hi, >> > > > >> > > > I would like to clarify migration path to 5.0.0 >> from 4.X.X >> > clusters. >> > > > For all Spectrum Scale clusters that are >> currently at 4.X.X, >> > it is >> > > > possible to migrate to 5.0.0 with no offline >> data migration >> > and no >> > > > need to move data. Once these clusters are at >> 5.0.0, they will >> > > > benefit from the performance improvements, new >> features (such as >> > > > file audit logging), and various enhancements >> that are >> > included in >> > > 5.0.0. >> > > > >> > > > That being said, there is one enhancement that >> will not be >> > applied >> > > > to these clusters, and that is the increased >> number of >> > sub-blocks >> > > > per block for small file allocation. This means >> that for file >> > > > systems with a large block size and a lot of >> small files, the >> > > > overall space utilization will be the same it >> currently is >> > in 4.X.X. >> > > > Since file systems created at 4.X.X and earlier >> used a block >> > size >> > > > that kept this allocation in mind, there should >> be very little >> > > > impact on existing file systems. >> > > > >> > > > Outside of that one particular function, the >> remainder of the >> > > > performance improvements, metadata improvements, >> updated >> > > > compatibility, new functionality, and all of the >> other >> > enhancements >> > > > will be immediately available to you once you >> complete the >> > upgrade >> > > > to 5.0.0 -- with no need to reformat, move data, >> or take >> > your data >> > > offline. >> > > > >> > > > I hope that clarifies things a little and makes >> the upgrade path >> > > > more accessible. >> > > > >> > > > Please let me know if there are any other >> questions or concerns. >> > > > >> > > > Thank you, >> > > > Nikhil Khandelwal >> > > > Spectrum Scale Development >> > > > Client Adoption >> > > > >> > > > _______________________________________________ >> > > > gpfsug-discuss mailing list >> > > > gpfsug-discuss at _spectrumscale.org_ >> > > >> > >> >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e= >> > >> >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>> >> > > > _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_ >> > <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_> >> > > >> > >> >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e= >> > >> >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>> >> > > >> > > > _______________________________________________ >> > > > gpfsug-discuss mailing list >> > > > gpfsug-discuss at _spectrumscale.org_ >> > > >> > >> >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e= >> > >> >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>> >> > > >> > > > _https://urldefense.proofpoint.com/v2/url?_ >> > <https://urldefense.proofpoint.com/v2/url?_> >> > > > >> > > >> >> u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- >> > > > >> > > >> >> siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e= >> > > >> > > _______________________________________________ >> > > gpfsug-discuss mailing list >> > > gpfsug-discuss at _spectrumscale.org_ >> > > >> > >> >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e= >> > >> >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>_ >> > > __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_ >> > <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_> >> > > >> > >> >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e= >> > >> >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>_______________________________________________ >> > > gpfsug-discuss mailing list >> > > gpfsug-discuss at spectrumscale.org >> <http://spectrumscale.org> <http://spectrumscale.org> >> > > >> > >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e= >> > >> >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=> >> > > >> > > >> > > >> > > >> > > _______________________________________________ >> > > gpfsug-discuss mailing list >> > > gpfsug-discuss at spectrumscale.org >> <http://spectrumscale.org> <http://spectrumscale.org> >> > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > <http://gpfsug.org/mailman/listinfo/gpfsug-discuss> >> > > >> > >> > -- >> > Aaron Knister >> > NASA Center for Climate Simulation (Code 606.2) >> > Goddard Space Flight Center >> > (301) 286-2776 <tel:(301)%20286-2776> >> <tel:%28301%29%20286-2776> >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> <http://spectrumscale.org> <http://spectrumscale.org> >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > <http://gpfsug.org/mailman/listinfo/gpfsug-discuss> >> > >> > >> > >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org <http://spectrumscale.org> >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 <tel:(301)%20286-2776> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org <http://spectrumscale.org> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org <http://spectrumscale.org> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >
-- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
