Re: [gpfsug-discuss] AFM Alternative? Aspera?

[Venkateswara R Puvvada]

Transferring the small files with AFM  + NFS over high latency networks is 
 always a challenge. For example,  for each small file replication AFM 
performs a lookup, create, write and set mtime operation. If the latency 
is 10ms,  replication of each file takes minimum (10 * 4 = 40 ms)  amount 
of time. AFM is not a network acceleration tool and also it does not use 
compression.  If the file sizes are big, AFM parallel IO and parallel 
mounts feature can be used.  Aspera can be used to transfer the small 
files over high latency network with better utilization of the network 
bandwidth.

https://www.ibm.com/support/knowledgecenter/no/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/b1lins_afmparalleldatatransferwithremotemounts.htm
https://www.ibm.com/support/knowledgecenter/no/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/bl1ins_paralleldatatransfersafm.htm

~Venkat (vpuvv...@in.ibm.com)



From:   Chris Schlipalius 
To: 
Date:   02/27/2020 05:54 AM
Subject:[EXTERNAL] Re: [gpfsug-discuss] AFM Alternative? Aspera?
Sent by:gpfsug-discuss-boun...@spectrumscale.org



Maybe the following would assist? I do think tarring up files first is 
best, but you could always check out:
http://www.redbooks.ibm.com/redpapers/pdfs/redp5527.pdf
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.spectrumscaleug.org_wp-2Dcontent_uploads_2019_05_SSSD19DE-2DDay-2D2-2DB02-2DIntegration-2Dof-2DSpectrum-2DScale-2Dand-2DAspera-2DSync.pdf=DwIGaQ=jf_iaSHvJObTbx-siA1ZOg=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A=1pVcjKeZ7gCaDtLoJFbfKCETe1XOmol6d2ryoccqC1A=tRCxd4SimJH_eycqekhzM0Qp3TB3NtaIYWBvyQnrIiM=
 


Aspera sync integration

(non html links added for your use – how they don’t get scrubbed:
www.spectrumscaleug.org/wp-content/uploads/2019/05/SSSD19DE-Day-2-B02-Integration-of-Spectrum-Scale-and-Aspera-Sync.pdf
 

www.redbooks.ibm.com/redpapers/pdfs/redp5527.pdf
)
Regards,
Chris Schlipalius
 
Team Lead, Data Storage Infrastructure, Data & Visualisation, Pawsey 
Supercomputing Centre (CSIRO)
1 Bryce Avenue
Kensington  WA  6151
Australia
 
Tel  +61 8 6436 8815 
Email  chris.schlipal...@pawsey.org.au
Web  www.pawsey.org.au <
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pawsey.org.au_=DwIGaQ=jf_iaSHvJObTbx-siA1ZOg=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A=1pVcjKeZ7gCaDtLoJFbfKCETe1XOmol6d2ryoccqC1A=Xkm8VFy3l6nyD40yhONihsKcqmwRhy4SZyd0lwHf1GA=
 
>
 
 
 


On 26/2/20, 9:39 pm, "gpfsug-discuss-boun...@spectrumscale.org on behalf 
of gpfsug-discuss-requ...@spectrumscale.org" 
 wrote:

Re: AFM Alternative?



___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss=DwIGaQ=jf_iaSHvJObTbx-siA1ZOg=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A=1pVcjKeZ7gCaDtLoJFbfKCETe1XOmol6d2ryoccqC1A=mYK1ZsVgtsM6HntRMLPS49tKvEhhgGAdWF2qniyn9Ko=
 






___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] AFM Alternative? Aspera?

[Chris Schlipalius]

Maybe the following would assist? I do think tarring up files first is best, 
but you could always check out:
http://www.redbooks.ibm.com/redpapers/pdfs/redp5527.pdf
https://www.spectrumscaleug.org/wp-content/uploads/2019/05/SSSD19DE-Day-2-B02-Integration-of-Spectrum-Scale-and-Aspera-Sync.pdf

Aspera sync integration

(non html links added for your use – how they don’t get scrubbed:
www.spectrumscaleug.org/wp-content/uploads/2019/05/SSSD19DE-Day-2-B02-Integration-of-Spectrum-Scale-and-Aspera-Sync.pdf
 
www.redbooks.ibm.com/redpapers/pdfs/redp5527.pdf
)
Regards,
Chris Schlipalius
 
Team Lead, Data Storage Infrastructure, Data & Visualisation, Pawsey 
Supercomputing Centre (CSIRO)
1 Bryce Avenue
Kensington  WA  6151
Australia
 
Tel  +61 8 6436 8815 
Email  chris.schlipal...@pawsey.org.au
Web  www.pawsey.org.au 
 
 
 


On 26/2/20, 9:39 pm, "gpfsug-discuss-boun...@spectrumscale.org on behalf of 
gpfsug-discuss-requ...@spectrumscale.org" 
 wrote:

Re: AFM Alternative?



___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] AFM Alternative?

[Sven Oehme]

if you are looking for a commercial supported solution, our Dataflow
product is purpose build for this kind of task. a presentation that
covers some high level aspects of it was given by me last year at one
of the spectrum scale meetings in the UK -->
https://www.spectrumscaleug.org/wp-content/uploads/2019/05/SSUG19UK-Day-1-05-DDN-Optimizing-storage-stacks-for-AI.pdf.
its at the end of the deck.

if you want more infos, please let me know and i can get you in
contact with the right person. Sven

On Wed, Feb 26, 2020 at 7:24 AM Frederick Stock  wrote:
>
> What sources are you using to help you with configuring AFM?
>
> Fred
> __
> Fred Stock | IBM Pittsburgh Lab | 720-430-8821
> sto...@us.ibm.com
>
>
>
> - Original message -
> From: Andi Christiansen 
> To: Frederick Stock , gpfsug-discuss@spectrumscale.org
> Cc:
> Subject: [EXTERNAL] RE: [gpfsug-discuss] AFM Alternative?
> Date: Wed, Feb 26, 2020 8:39 AM
>
> 5.0.4-2.1 (home and cache)
>
> On February 26, 2020 2:33 PM Frederick Stock  wrote:
>
>
> Andi, what version of Spectrum Scale do you have installed?
>
> Fred
> __
> Fred Stock | IBM Pittsburgh Lab | 720-430-8821
> sto...@us.ibm.com
>
>
>
> - Original message -
> From: "Olaf Weiser" 
> Sent by: gpfsug-discuss-boun...@spectrumscale.org
> To: a...@christiansen.xxx, gpfsug-discuss@spectrumscale.org
> Cc: gpfsug-discuss@spectrumscale.org
> Subject: [EXTERNAL] Re: [gpfsug-discuss] AFM Alternative?
> Date: Wed, Feb 26, 2020 8:27 AM
>
> you may consider WatchFolder  ... (cluster wider inotify --> kafka) .. and 
> then you go from there
>
>
>
> - Original message -
> From: Andi Christiansen 
> Sent by: gpfsug-discuss-boun...@spectrumscale.org
> To: "gpfsug-discuss@spectrumscale.org" 
> Cc:
> Subject: [EXTERNAL] [gpfsug-discuss] AFM Alternative?
> Date: Wed, Feb 26, 2020 1:59 PM
>
> Hi all,
>
> Does anyone know of an alternative to AFM ?
>
> We have been working on tuning AFM for a few weeks now and see little to no 
> improvement.. And now we are searching for an alternative.. So if anyone 
> knows of a product that can implement with Spectrum Scale i am open to any 
> suggestions :)
>
> We have a good mix of files but primarily billions of very small files which 
> AFM does not handle well on long distances.
>
>
> Best Regards
> A. Christiansen
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
>
>
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] AFM Alternative?

[Frederick Stock]

What sources are you using to help you with configuring AFM?
Fred__Fred Stock | IBM Pittsburgh Lab | 720-430-8821sto...@us.ibm.com
 
 
- Original message -From: Andi Christiansen To: Frederick Stock , gpfsug-discuss@spectrumscale.orgCc:Subject: [EXTERNAL] RE: [gpfsug-discuss] AFM Alternative?Date: Wed, Feb 26, 2020 8:39 AM 
5.0.4-2.1 (home and cache)
On February 26, 2020 2:33 PM Frederick Stock  wrote:
 
 
Andi, what version of Spectrum Scale do you have installed?
Fred__Fred Stock | IBM Pittsburgh Lab | 720-430-8821sto...@us.ibm.com
 
 
- Original message -From: "Olaf Weiser" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: a...@christiansen.xxx, gpfsug-discuss@spectrumscale.orgCc: gpfsug-discuss@spectrumscale.orgSubject: [EXTERNAL] Re: [gpfsug-discuss] AFM Alternative?Date: Wed, Feb 26, 2020 8:27 AM 
you may consider WatchFolder  ... (cluster wider inotify --> kafka) .. and then you go from there
 
 
- Original message -From: Andi Christiansen Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: "gpfsug-discuss@spectrumscale.org" Cc:Subject: [EXTERNAL] [gpfsug-discuss] AFM Alternative?Date: Wed, Feb 26, 2020 1:59 PM 
Hi all,
 
Does anyone know of an alternative to AFM ?
 
We have been working on tuning AFM for a few weeks now and see little to no improvement.. And now we are searching for an alternative.. So if anyone knows of a product that can implement with Spectrum Scale i am open to any suggestions :)
 
We have a good mix of files but primarily billions of very small files which AFM does not handle well on long distances.
 
 
Best Regards
A. Christiansen
___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss 
  

___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss 
 
 

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] AFM Alternative?

[Andi Christiansen]

 
 
  
   Hmm.. i dont know what that is! i will have to look into that! Thanks! :) 
  
  
   
On February 26, 2020 2:27 PM Olaf Weiser  wrote:
   
   

   
   

   
   

 you may consider WatchFolder  ... (cluster wider inotify --> kafka) .. and then you go from there


 


 


 - Original message -
 From: Andi Christiansen 
 Sent by: gpfsug-discuss-boun...@spectrumscale.org
 To: "gpfsug-discuss@spectrumscale.org" 
 Cc:
 Subject: [EXTERNAL] [gpfsug-discuss] AFM Alternative?
 Date: Wed, Feb 26, 2020 1:59 PM
  
 
  Hi all,
 
 
  
 
 
  Does anyone know of an alternative to AFM ?
 
 
  
 
 
  We have been working on tuning AFM for a few weeks now and see little to no improvement.. And now we are searching for an alternative.. So if anyone knows of a product that can implement with Spectrum Scale i am open to any suggestions :)
 
 
  
 
 
  We have a good mix of files but primarily billions of very small files which AFM does not handle well on long distances.
 
 
  
 
 
  
 
 
  Best Regards
 
 
  A. Christiansen
 
 
  ___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss 
 


 

   
   
   
 

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] AFM Alternative?

[Andi Christiansen]

 
 
  
   5.0.4-2.1 (home and cache)
  
  
   
On February 26, 2020 2:33 PM Frederick Stock  wrote:
   
   

   
   

   
   

 Andi, what version of Spectrum Scale do you have installed?


 
  
   
   Fred__Fred Stock | IBM Pittsburgh Lab | 720-430-8821sto...@us.ibm.com
  
 


 


 


 - Original message -
 From: "Olaf Weiser" 
 Sent by: gpfsug-discuss-boun...@spectrumscale.org
 To: a...@christiansen.xxx, gpfsug-discuss@spectrumscale.org
 Cc: gpfsug-discuss@spectrumscale.org
 Subject: [EXTERNAL] Re: [gpfsug-discuss] AFM Alternative?
 Date: Wed, Feb 26, 2020 8:27 AM
  
 
  
   you may consider WatchFolder  ... (cluster wider inotify --> kafka) .. and then you go from there
  
  
   
  
  
   
  
  
   - Original message -
   From: Andi Christiansen 
   Sent by: gpfsug-discuss-boun...@spectrumscale.org
   To: "gpfsug-discuss@spectrumscale.org" 
   Cc:
   Subject: [EXTERNAL] [gpfsug-discuss] AFM Alternative?
   Date: Wed, Feb 26, 2020 1:59 PM
    
   
Hi all,
   
   

   
   
Does anyone know of an alternative to AFM ?
   
   

   
   
We have been working on tuning AFM for a few weeks now and see little to no improvement.. And now we are searching for an alternative.. So if anyone knows of a product that can implement with Spectrum Scale i am open to any suggestions :)
   
   

   
   
We have a good mix of files but primarily billions of very small files which AFM does not handle well on long distances.
   
   

   
   

   
   
Best Regards
   
   
A. Christiansen
   
   
___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss 
   
  
  
   
  
  
 
  ___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss 
 


 

   
   
   
 

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] AFM Alternative?

[Andi Christiansen]

 
 
  
   i seem to have read somewhere that packing is just wasting time on zipping as the same file will be transfered so i havent really looked to much into that? and i dont think that is even possible in our use case as that would also mean that we in theory need to store the same data twice at source site (unzipped and zipped data).
  
  
   
  
  
   Best Regards
   Andi Christiansen
  
  
   
On February 26, 2020 2:04 PM Andrew Beattie  wrote:
   
   

   
   

   
   Why don’t you look at packaging your small files into larger files which will be handled more effectively.There is no simple way to replicate / move billions of small files,But surely you can build your work flow to package the files up into a zip or tar format which will simplify not only the number of IO transactions but also make the whole process more palatable to the NFS protocolSent from my iPhone> On 26 Feb 2020, at 22:58, Andi Christiansen  wrote:> > > Hi all,> > Does anyone know of an alternative to AFM ?> > We have been working on tuning AFM for a few weeks now and see little to no improvement.. And now we are searching for an alternative.. So if anyone knows of a product that can implement with Spectrum Scale i am open to any suggestions :)> > We have a good mix of files but primarily billions of very small files which AFM does not handle well on long distances.> > > Best Regards> A. Christiansen> ___> gpfsug-discuss mailing list> gpfsug-discuss at spectrumscale.org> http://gpfsug.org/mailman/listinfo/gpfsug-discuss> 
   
 

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] AFM Alternative?

[Frederick Stock]

Andi, what version of Spectrum Scale do you have installed?
Fred__Fred Stock | IBM Pittsburgh Lab | 720-430-8821sto...@us.ibm.com
 
 
- Original message -From: "Olaf Weiser" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: a...@christiansen.xxx, gpfsug-discuss@spectrumscale.orgCc: gpfsug-discuss@spectrumscale.orgSubject: [EXTERNAL] Re: [gpfsug-discuss] AFM Alternative?Date: Wed, Feb 26, 2020 8:27 AM 
you may consider WatchFolder  ... (cluster wider inotify --> kafka) .. and then you go from there
 
 
- Original message -From: Andi Christiansen Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: "gpfsug-discuss@spectrumscale.org" Cc:Subject: [EXTERNAL] [gpfsug-discuss] AFM Alternative?Date: Wed, Feb 26, 2020 1:59 PM 
Hi all,
 
Does anyone know of an alternative to AFM ?
 
We have been working on tuning AFM for a few weeks now and see little to no improvement.. And now we are searching for an alternative.. So if anyone knows of a product that can implement with Spectrum Scale i am open to any suggestions :)
 
We have a good mix of files but primarily billions of very small files which AFM does not handle well on long distances.
 
 
Best Regards
A. Christiansen
___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss 
  

___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss 
 

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] AFM Alternative?

[Olaf Weiser]

you may consider WatchFolder  ... (cluster wider inotify --> kafka) .. and then you go from there
 
 
- Original message -From: Andi Christiansen Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: "gpfsug-discuss@spectrumscale.org" Cc:Subject: [EXTERNAL] [gpfsug-discuss] AFM Alternative?Date: Wed, Feb 26, 2020 1:59 PM 
Hi all,
 
Does anyone know of an alternative to AFM ?
 
We have been working on tuning AFM for a few weeks now and see little to no improvement.. And now we are searching for an alternative.. So if anyone knows of a product that can implement with Spectrum Scale i am open to any suggestions :)
 
We have a good mix of files but primarily billions of very small files which AFM does not handle well on long distances.
 
 
Best Regards
A. Christiansen
___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss 
 

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] AFM Alternative?

[Andrew Beattie]

Why don’t you look at packaging your small files into larger files which
will be handled more effectively.

There is no simple way to replicate / move billions of small files,

But surely you can build your work flow to package the files up into a zip
or tar format which will simplify not only the number of IO transactions
but also make the whole process more palatable to the NFS protocol

Sent from my iPhone

> On 26 Feb 2020, at 22:58, Andi Christiansen 
wrote:
>
> 
> Hi all,
>
> Does anyone know of an alternative to AFM ?
>
> We have been working on tuning AFM for a few weeks now and see little to
no improvement.. And now we are searching for an alternative.. So if anyone
knows of a product that can implement with Spectrum Scale i am open to any
suggestions :)
>
> We have a good mix of files but primarily billions of very small files
which AFM does not handle well on long distances.
>
>
> Best Regards
> A. Christiansen
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
>
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss=DwICAg=jf_iaSHvJObTbx-siA1ZOg=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk=BDsYqP0is2zoDGYU5Ej1lSJ4s9DJhMsW40equi5dqCs=22KcLJbUqsq3nfr3qWnxDqA3kuHnFxSDeiENVUITmdA=

>
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

[gpfsug-discuss] AFM Alternative?

[Andi Christiansen]

 
 
  
   Hi all,
  
  
   
  
  
   Does anyone know of an alternative to AFM ?
  
  
   
  
  
   We have been working on tuning AFM for a few weeks now and see little to no improvement.. And now we are searching for an alternative.. So if anyone knows of a product that can implement with Spectrum Scale i am open to any suggestions :)
  
  
   
  
  
   We have a good mix of files but primarily billions of very small files which AFM does not handle well on long distances.
  
  
   
  
  
   
  
  
   Best Regards
  
  
   A. Christiansen
   
 

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss