Re: [gpfsug-discuss] Exporting remote GPFS mounts on a non-ces SMB share

[valleru]

Thanks a lot Andrew.

It does look promising but It does not strike me immediately on how this could 
solve the SMB export where user authenticates with an AD username but the gpfs 
files that are present are owned by LDAP username.
May be you are saying that if i enable GPFS to use these scripts - then GPFS 
will map the AD username to the LDAP username?

I found this url too..

https://www.ibm.com/support/knowledgecenter/en/SSFKCN/com.ibm.cluster.gpfs.doc/gpfs_uid/uid_gpfs.html

I will give it a read, try to understand how to implement it and get back if i 
have any more questions.

If this works, it should help me configure and use the CES SMB. (Hopefully, CES 
file based authentication will allow both ssh key authentication for NFS and AD 
for SMB in same CES cluster).

Regards,
Lohit

On Mar 7, 2019, 4:52 PM -0600, Andrew Beattie , wrote:
> Lohit
>
> Have you looked at mmUIDtoName mmNametoUID
>
> Yes it will require some custom scripting on your behalf but it would be a 
> far more elegant solution and not run the risk of data corruption issues.
>
> There is at least one university on this mailing list that is doing exactly 
> what you are talking about, and they successfully use
> mmUIDtoName / mmNametoUID  to provide the relevant mapping between different 
> authentication environments - both internally in the university and 
> externally from other institutions.
>
> They use AFM to move data between different storage clusters, and mmUIDtoName 
> / mmNametoUID, to manage the ACL and permissions, they then move the data 
> from the AFM filesystem to the HPC scratch filesystem for processing by the 
> HPC (different filesystems within the same cluster)
>
>
> Regards,
> Andrew Beattie
> File and Object Storage Technical Specialist - A/NZ
> IBM Systems - Storage
> Phone: 614-2133-7927
> E-mail: abeat...@au1.ibm.com
>
>
> > - Original message -
> > From: vall...@cbio.mskcc.org
> > Sent by: gpfsug-discuss-boun...@spectrumscale.org
> > To: gpfsug-discuss@spectrumscale.org, gpfsug main discussion list 
> > 
> > Cc:
> > Subject: Re: [gpfsug-discuss] Exporting remote GPFS mounts on a non-ces SMB 
> > share
> > Date: Fri, Mar 8, 2019 8:21 AM
> >
> > We have many current usernames from LDAP that do not exactly match with the 
> > usernames from AD.
> > Unfortunately, i guess CES SMB will need us to use either AD or LDAP or use 
> > the same usernames in both AD and LDAP.
> > I have been looking for a solution where could map the different usernames 
> > from LDAP and AD but have not found a solution. So exploring ways to do 
> > this from RHEL SMB.
> > I would appreciate if you have any solution to this issue.
> >
> > As of now we use LDAP uids/gids and SSH keys for authentication to the HPC 
> > cluster.
> > We want to use CES SMB to export the same mounts which have LDAP 
> > usernames/uids/gids however because of different usernames in AD - it has 
> > become a challenge.
> > Even if we do find a solution to this, i want to be able to use AD 
> > authentication for SMB and ssh key authentication for NFS.
> >
> > The above are the reasons we are just using CES with NFS and user defined 
> > authentication for users to have access with login through ssh keys.
> >
> > Regards,
> > Lohit
> >
> > On Mar 7, 2019, 3:12 PM -0600, Andrew Beattie , wrote:
> > > That would not be supported
> > >
> > > You shouldn't publish a remote mount Protocol cluster , and then connect 
> > > a native client to that cluster and create a non CES protocol export
> > > if you are going to use a Protocol cluster that's how you present your 
> > > protocols.
> > > otherwise don't set up the remote mount cluster.
> > >
> > > Why are you trying to publish a non HA RHEL SMB share instead of using 
> > > the HA CES protocols?
> > > Andrew Beattie
> > > File and Object Storage Technical Specialist - A/NZ
> > > IBM Systems - Storage
> > > Phone: 614-2133-7927
> > > E-mail: abeat...@au1.ibm.com
> > >
> > >
> > > > - Original message -
> > > > From: vall...@cbio.mskcc.org
> > > > Sent by: gpfsug-discuss-boun...@spectrumscale.org
> > > > To: gpfsug-discuss@spectrumscale.org, gpfsug main discussion list 
> > > > 
> > > > Cc:
> > > > Subject: Re: [gpfsug-discuss] Exporting remote GPFS mounts on a non-ces 
> > > > SMB share
> > > > Date: Fri, Mar 8, 2019 7:05 AM
> > > >
> > > > Thank you Andrew.
> > > >
> > > > However, we are not using SMB from the CES cluster but instead running 
> > > > a Redhat based SMB on a GPFS client of the CES cluster and exporting it 
> > > > from the GPFS client.
> > > > Is the above supported, and not known to cause any issues?
> > > >
> > > > Regards,
> > > > Lohit
> > > >
> > > > On Mar 7, 2019, 2:45 PM -0600, Andrew Beattie , 
> > > > wrote:
> > > > >
> > > > > https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.2/com.ibm.spectrum.scale.v5r02.doc/bl1adv_configprotocolsonremotefs.htm
> > > > ___
> > > > gpfsug-discuss mailing list
> > > > 

Re: [gpfsug-discuss] Exporting remote GPFS mounts on a non-ces SMB share

[Simon Thompson]

There is a custom Auth mode I think that allows you to use ad for Auth and LDAP 
for identity. You'd could do what you wanted but you'd need another LDAP 
instance that mapped the ad usernames to the UID that is only used by SMB. 

Hack yes.

Simon

From: gpfsug-discuss-boun...@spectrumscale.org 
[gpfsug-discuss-boun...@spectrumscale.org] on behalf of vall...@cbio.mskcc.org 
[vall...@cbio.mskcc.org]
Sent: 07 March 2019 22:10
To: gpfsug-discuss@spectrumscale.org; gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Exporting remote GPFS mounts on a non-ces SMB 
share

We have many current usernames from LDAP that do not exactly match with the 
usernames from AD.
Unfortunately, i guess CES SMB will need us to use either AD or LDAP or use the 
same usernames in both AD and LDAP.
I have been looking for a solution where could map the different usernames from 
LDAP and AD but have not found a solution. So exploring ways to do this from 
RHEL SMB.
I would appreciate if you have any solution to this issue.

As of now we use LDAP uids/gids and SSH keys for authentication to the HPC 
cluster.
We want to use CES SMB to export the same mounts which have LDAP 
usernames/uids/gids however because of different usernames in AD - it has 
become a challenge.
Even if we do find a solution to this, i want to be able to use AD 
authentication for SMB and ssh key authentication for NFS.

The above are the reasons we are just using CES with NFS and user defined 
authentication for users to have access with login through ssh keys.

Regards,
Lohit

On Mar 7, 2019, 3:12 PM -0600, Andrew Beattie , wrote:
That would not be supported

You shouldn't publish a remote mount Protocol cluster , and then connect a 
native client to that cluster and create a non CES protocol export
if you are going to use a Protocol cluster that's how you present your 
protocols.
otherwise don't set up the remote mount cluster.

Why are you trying to publish a non HA RHEL SMB share instead of using the HA 
CES protocols?
Andrew Beattie
File and Object Storage Technical Specialist - A/NZ
IBM Systems - Storage
Phone: 614-2133-7927
E-mail: abeat...@au1.ibm.com


- Original message -
From: vall...@cbio.mskcc.org
Sent by: gpfsug-discuss-boun...@spectrumscale.org
To: gpfsug-discuss@spectrumscale.org, gpfsug main discussion list 

Cc:
Subject: Re: [gpfsug-discuss] Exporting remote GPFS mounts on a non-ces SMB 
share
Date: Fri, Mar 8, 2019 7:05 AM

Thank you Andrew.

However, we are not using SMB from the CES cluster but instead running a Redhat 
based SMB on a GPFS client of the CES cluster and exporting it from the GPFS 
client.
Is the above supported, and not known to cause any issues?

Regards,
Lohit

On Mar 7, 2019, 2:45 PM -0600, Andrew Beattie , wrote:

https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.2/com.ibm.spectrum.scale.v5r02.doc/bl1adv_configprotocolsonremotefs.htm
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Exporting remote GPFS mounts on a non-ces SMB share

[Andrew Beattie]

Lohit
 
Have you looked at mmUIDtoName mmNametoUID
 
Yes it will require some custom scripting on your behalf but it would be a far more elegant solution and not run the risk of data corruption issues.
 
There is at least one university on this mailing list that is doing exactly what you are talking about, and they successfully use
mmUIDtoName / mmNametoUID  to provide the relevant mapping between different authentication environments - both internally in the university and externally from other institutions.
 
They use AFM to move data between different storage clusters, and mmUIDtoName / mmNametoUID, to manage the ACL and permissions, they then move the data from the AFM filesystem to the HPC scratch filesystem for processing by the HPC (different filesystems within the same cluster)
 
 
Regards,
Andrew Beattie
File and Object Storage Technical Specialist - A/NZ
IBM Systems - Storage
Phone: 614-2133-7927
E-mail: abeat...@au1.ibm.com
 
 
- Original message -From: vall...@cbio.mskcc.orgSent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug-discuss@spectrumscale.org, gpfsug main discussion list Cc:Subject: Re: [gpfsug-discuss] Exporting remote GPFS mounts on a non-ces SMB shareDate: Fri, Mar 8, 2019 8:21 AM 
We have many current usernames from LDAP that do not exactly match with the usernames from AD.
Unfortunately, i guess CES SMB will need us to use either AD or LDAP or use the same usernames in both AD and LDAP.
I have been looking for a solution where could map the different usernames from LDAP and AD but have not found a solution. So exploring ways to do this from RHEL SMB.
I would appreciate if you have any solution to this issue.
 
As of now we use LDAP uids/gids and SSH keys for authentication to the HPC cluster. 
We want to use CES SMB to export the same mounts which have LDAP usernames/uids/gids however because of different usernames in AD - it has become a challenge.
Even if we do find a solution to this, i want to be able to use AD authentication for SMB and ssh key authentication for NFS.
 
The above are the reasons we are just using CES with NFS and user defined authentication for users to have access with login through ssh keys.
Regards,Lohit
On Mar 7, 2019, 3:12 PM -0600, Andrew Beattie , wrote:
That would not be supported
 
You shouldn't publish a remote mount Protocol cluster , and then connect a native client to that cluster and create a non CES protocol export
if you are going to use a Protocol cluster that's how you present your protocols.
otherwise don't set up the remote mount cluster.
 
Why are you trying to publish a non HA RHEL SMB share instead of using the HA CES protocols?
Andrew Beattie
File and Object Storage Technical Specialist - A/NZ
IBM Systems - Storage
Phone: 614-2133-7927
E-mail: abeat...@au1.ibm.com
 
 
- Original message -From: vall...@cbio.mskcc.orgSent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug-discuss@spectrumscale.org, gpfsug main discussion list Cc:Subject: Re: [gpfsug-discuss] Exporting remote GPFS mounts on a non-ces SMB shareDate: Fri, Mar 8, 2019 7:05 AM 
Thank you Andrew.
 
However, we are not using SMB from the CES cluster but instead running a Redhat based SMB on a GPFS client of the CES cluster and exporting it from the GPFS client.
Is the above supported, and not known to cause any issues?
Regards,Lohit
On Mar 7, 2019, 2:45 PM -0600, Andrew Beattie , wrote:
 
https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.2/com.ibm.spectrum.scale.v5r02.doc/bl1adv_configprotocolsonremotefs.htm
___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss
 ___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss
___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss
 

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Exporting remote GPFS mounts on a non-ces SMB share

[valleru]

We have many current usernames from LDAP that do not exactly match with the 
usernames from AD.
Unfortunately, i guess CES SMB will need us to use either AD or LDAP or use the 
same usernames in both AD and LDAP.
I have been looking for a solution where could map the different usernames from 
LDAP and AD but have not found a solution. So exploring ways to do this from 
RHEL SMB.
I would appreciate if you have any solution to this issue.

As of now we use LDAP uids/gids and SSH keys for authentication to the HPC 
cluster.
We want to use CES SMB to export the same mounts which have LDAP 
usernames/uids/gids however because of different usernames in AD - it has 
become a challenge.
Even if we do find a solution to this, i want to be able to use AD 
authentication for SMB and ssh key authentication for NFS.

The above are the reasons we are just using CES with NFS and user defined 
authentication for users to have access with login through ssh keys.

Regards,
Lohit

On Mar 7, 2019, 3:12 PM -0600, Andrew Beattie , wrote:
> That would not be supported
>
> You shouldn't publish a remote mount Protocol cluster , and then connect a 
> native client to that cluster and create a non CES protocol export
> if you are going to use a Protocol cluster that's how you present your 
> protocols.
> otherwise don't set up the remote mount cluster.
>
> Why are you trying to publish a non HA RHEL SMB share instead of using the HA 
> CES protocols?
> Andrew Beattie
> File and Object Storage Technical Specialist - A/NZ
> IBM Systems - Storage
> Phone: 614-2133-7927
> E-mail: abeat...@au1.ibm.com
>
>
> > - Original message -
> > From: vall...@cbio.mskcc.org
> > Sent by: gpfsug-discuss-boun...@spectrumscale.org
> > To: gpfsug-discuss@spectrumscale.org, gpfsug main discussion list 
> > 
> > Cc:
> > Subject: Re: [gpfsug-discuss] Exporting remote GPFS mounts on a non-ces SMB 
> > share
> > Date: Fri, Mar 8, 2019 7:05 AM
> >
> > Thank you Andrew.
> >
> > However, we are not using SMB from the CES cluster but instead running a 
> > Redhat based SMB on a GPFS client of the CES cluster and exporting it from 
> > the GPFS client.
> > Is the above supported, and not known to cause any issues?
> >
> > Regards,
> > Lohit
> >
> > On Mar 7, 2019, 2:45 PM -0600, Andrew Beattie , wrote:
> > >
> > > https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.2/com.ibm.spectrum.scale.v5r02.doc/bl1adv_configprotocolsonremotefs.htm
> > ___
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Exporting remote GPFS mounts on a non-ces SMB share

[Andrew Beattie]

That would not be supported
 
You shouldn't publish a remote mount Protocol cluster , and then connect a native client to that cluster and create a non CES protocol export
if you are going to use a Protocol cluster that's how you present your protocols.
otherwise don't set up the remote mount cluster.
 
Why are you trying to publish a non HA RHEL SMB share instead of using the HA CES protocols?
Andrew Beattie
File and Object Storage Technical Specialist - A/NZ
IBM Systems - Storage
Phone: 614-2133-7927
E-mail: abeat...@au1.ibm.com
 
 
- Original message -From: vall...@cbio.mskcc.orgSent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug-discuss@spectrumscale.org, gpfsug main discussion list Cc:Subject: Re: [gpfsug-discuss] Exporting remote GPFS mounts on a non-ces SMB shareDate: Fri, Mar 8, 2019 7:05 AM 
Thank you Andrew.
 
However, we are not using SMB from the CES cluster but instead running a Redhat based SMB on a GPFS client of the CES cluster and exporting it from the GPFS client.
Is the above supported, and not known to cause any issues?
Regards,Lohit
On Mar 7, 2019, 2:45 PM -0600, Andrew Beattie , wrote:
 
https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.2/com.ibm.spectrum.scale.v5r02.doc/bl1adv_configprotocolsonremotefs.htm
___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss
 

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Exporting remote GPFS mounts on a non-ces SMB share

[valleru]

Thank you Andrew.

However, we are not using SMB from the CES cluster but instead running a Redhat 
based SMB on a GPFS client of the CES cluster and exporting it from the GPFS 
client.
Is the above supported, and not known to cause any issues?

Regards,
Lohit

On Mar 7, 2019, 2:45 PM -0600, Andrew Beattie , wrote:
>
> https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.2/com.ibm.spectrum.scale.v5r02.doc/bl1adv_configprotocolsonremotefs.htm
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Exporting remote GPFS mounts on a non-ces SMB share

[Andrew Beattie]

Lohit,
 
What you are trying to achieve is a supported architecture, there are several limitations however
 
This is the list of limitations on setting up remote mount protocols
https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.2/com.ibm.spectrum.scale.v5r02.doc/bl1adv_limitationofprotocolonRMT.htm
 
This is the commended architecture
https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.2/com.ibm.spectrum.scale.v5r02.doc/bl1adv_protocoloverremoteclu.htm
 
This is the actual procedure
 
https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.2/com.ibm.spectrum.scale.v5r02.doc/bl1adv_configprotocolsonremotefs.htm
 
 
Hope this helps,
 
Andrew Beattie
File and Object Storage Technical Specialist - A/NZ
IBM Systems - Storage
Phone: 614-2133-7927
E-mail: abeat...@au1.ibm.com
 
 
- Original message -From: vall...@cbio.mskcc.orgSent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug main discussion list Cc:Subject: [gpfsug-discuss] Exporting remote GPFS mounts on a non-ces SMB shareDate: Fri, Mar 8, 2019 5:17 AM 
Hello All,
 
We are thinking of exporting “remote" GPFS mounts on a remote GPFS 5.0 cluster through a SMB share.
 
I have heard in a previous thread that it is not a good idea to export NFS/SMB share on a remote GPFS mount, and make it writable.
 
The issue that could be caused by making it writable would be metanode swapping between the GPFS clusters.
 
May i understand this better and the seriousness of this issue?
 
The possibility of a single file being written at the same time from a GPFS node and NFS/SMB node is minimum - however it is possible that a file is written at the same time from multiple protocols by mistake and we cannot prevent it.
 
This is the setup:
 
GPFS storage cluster: /gpfs01
GPFS CES cluster ( does not have any storage) : /gpfs01 -> mounted remotely . NFS export /gpfs01 as part of CES cluster
GPFS client for CES cluster -> Acts as SMB server and exports /gpfs01 over SMB
 
Are there any other limitations that i need to know for the above setup?
 
We cannot use GPFS CES SMB as of now for few other reasons such as LDAP/AD id mapping and authentication complications.
Regards,Lohit
___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss
 

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

[gpfsug-discuss] Exporting remote GPFS mounts on a non-ces SMB share

[valleru]

Hello All,

We are thinking of exporting “remote" GPFS mounts on a remote GPFS 5.0 cluster 
through a SMB share.

I have heard in a previous thread that it is not a good idea to export NFS/SMB 
share on a remote GPFS mount, and make it writable.

The issue that could be caused by making it writable would be metanode swapping 
between the GPFS clusters.

May i understand this better and the seriousness of this issue?

The possibility of a single file being written at the same time from a GPFS 
node and NFS/SMB node is minimum - however it is possible that a file is 
written at the same time from multiple protocols by mistake and we cannot 
prevent it.

This is the setup:

GPFS storage cluster: /gpfs01
GPFS CES cluster ( does not have any storage) : /gpfs01 -> mounted remotely . 
NFS export /gpfs01 as part of CES cluster
GPFS client for CES cluster -> Acts as SMB server and exports /gpfs01 over SMB

Are there any other limitations that i need to know for the above setup?

We cannot use GPFS CES SMB as of now for few other reasons such as LDAP/AD id 
mapping and authentication complications.

Regards,
Lohit
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Follow-up: migrating billions of files

[Jonathan Buzzard]

On Wed, 2019-03-06 at 12:44 +, Oesterlin, Robert wrote:
> Some of you had questions to my original post. More information:
>  
> Source:
> - Files are straight GPFS/Posix - no extended NFSV4 ACLs
> - A solution that requires $’s to be spent on software (ie, Aspera)
> isn’t a very viable option
> - Both source and target clusters are in the same DC
> - Source is stand-alone NSD servers (bonded 10g-E) and 8gb FC SAN
> storage
> - Approx 40 file systems, a few large ones with 300M-400M files each,
> others smaller
> - no independent file sets
> - migration must pose minimal disruption to existing users
>  
> Target architecture is a small number of file systems (2-3) on ESS
> with independent filesets
> - Target (ESS) will have multiple 40gb-E links on each NSD server
> (GS4)
>  
> My current thinking is AFM with a pre-populate of the file space and
> switch the clients over to have them pull data they need (most of the
> data is older and less active) and them let AFM populate the rest in
> the background.
>  

As it's not been mentioned yet "dsmc restore" or equivalent depending
on your backup solution.

JAB.

-- 
Jonathan A. Buzzard Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG



___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Follow-up: migrating billions of files

[Uwe Falke]

As for "making sure a subjob doesn't finish right after you go home 
leaving a slot idle for several hours ". 
That's the reason for the masterscript / control script / whatever. 
There would be a list of directories sorted to decreasing size, 
the master script would have a counter for each participating source host 
(a semaphore) and start as many parallel copy jobs, each with the 
currently topmost directory in the list, removing that directory (best 
possibly to an intermediary "in-work" list), counting down the semaphore 
on each start , unless 0. As soon as a job returns successfully, count up 
the semaphore, and if >0, start the next job, and so on. I suppose you can 
easily run about 8 to 12  such jobs per server (maybe best to use 
dedicated source server - dest server pairs).  So, no worries about 
leaving at any time WRT jobs ending and idle job slots . 

of course, some precautions should be taken to ensure each job succeeds 
and gets repeated if not , and a lot of logging should take place to be 
sure you would know what's happened.


 
Mit freundlichen Grüßen / Kind regards

 
Dr. Uwe Falke
 
IT Specialist
High Performance Computing Services / Integrated Technology Services / 
Data Center Services
---
IBM Deutschland
Rathausstr. 7
09111 Chemnitz
Phone: +49 371 6978 2165
Mobile: +49 175 575 2877
E-Mail: uwefa...@de.ibm.com
---
IBM Deutschland Business & Technology Services GmbH / Geschäftsführung: 
Thomas Wolter, Sven Schooß
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, 
HRB 17122 




From:   Stephen Ulmer 
To: gpfsug main discussion list 
Date:   06/03/2019 16:55
Subject:Re: [gpfsug-discuss] Follow-up: migrating billions of 
files
Sent by:gpfsug-discuss-boun...@spectrumscale.org



In the case where tar -C doesn?t work, you can always use a subshell (I do 
this regularly):

tar -cf . | ssh someguy@otherhost "(cd targetdir; tar -xvf - )"

Only use -v on one end. :)

Also, for parallel work that?s not designed that way, don't underestimate 
the -P option to GNU and BSD xargs! With the amount of stuff to be copied, 
making sure a subjob doesn?t finish right after you go home leaving a slot 
idle for several hours is a medium deal.

In Bob?s case, however, treating it like a DR exercise where users 
"restore" their own files by accessing them (using AFM instead of HSM) is 
probably the most convenient.

-- 
Stephen



On Mar 6, 2019, at 8:13 AM, Uwe Falke  wrote:

Hi, in that case I'd open several tar pipes in parallel, maybe using 
directories carefully selected, like 

 tar -c  | ssh   "tar -x"

I am not quite sure whether "-C /" for tar works here ("tar -C / -x"), but 

along these lines might be a good efficient method. target_hosts should be 

all nodes haveing the target file system mounted, and you should start 
those pipes on the nodes with the source file system. 
It is best to start with the largest directories, and use some 
masterscript to start the tar pipes controlled by semaphores  to not 
overload anything. 



Mit freundlichen Grüßen / Kind regards


Dr. Uwe Falke

IT Specialist
High Performance Computing Services / Integrated Technology Services / 
Data Center Services
---
IBM Deutschland
Rathausstr. 7
09111 Chemnitz
Phone: +49 371 6978 2165
Mobile: +49 175 575 2877
E-Mail: uwefa...@de.ibm.com
---
IBM Deutschland Business & Technology Services GmbH / Geschäftsführung: 
Thomas Wolter, Sven Schooß
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, 
HRB 17122 




From:   "Oesterlin, Robert" 
To: gpfsug main discussion list 
Date:   06/03/2019 13:44
Subject:[gpfsug-discuss] Follow-up: migrating billions of files
Sent by:gpfsug-discuss-boun...@spectrumscale.org



Some of you had questions to my original post. More information:

Source:
- Files are straight GPFS/Posix - no extended NFSV4 ACLs
- A solution that requires $?s to be spent on software (ie, Aspera) isn?t 
a very viable option
- Both source and target clusters are in the same DC
- Source is stand-alone NSD servers (bonded 10g-E) and 8gb FC SAN storage
- Approx 40 file systems, a few large ones with 300M-400M files each, 
others smaller
- no independent file sets
- migration must pose minimal disruption to existing users

Target architecture is a small number of file systems (2-3) on ESS with 
independent filesets
- Target (ESS) will have multiple 40gb-E links on each NSD server (GS4)

My current 

Re: [gpfsug-discuss] Memory accounting for processes writing to GPFS

[Dorigo Alvise (PSI)]

Thanks to all for clarification.

   A


From: gpfsug-discuss-boun...@spectrumscale.org 
[gpfsug-discuss-boun...@spectrumscale.org] on behalf of Tomer Perry 
[t...@il.ibm.com]
Sent: Wednesday, March 06, 2019 2:14 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Memory accounting for processes writing to GPFS

It might be the case that AsynchronousFileChannelis actually doing mmap access 
to the files. Thus, the memory management will be completely different with 
GPFS in compare to local fs.

Regards,

Tomer Perry
Scalable I/O Development (Spectrum Scale)
email: t...@il.ibm.com
1 Azrieli Center, Tel Aviv 67021, Israel
Global Tel:+1 720 3422758
Israel Tel:  +972 3 9188625
Mobile: +972 52 2554625




From:Jim Doherty 
To:gpfsug main discussion list 
Date:06/03/2019 06:59
Subject:Re: [gpfsug-discuss] Memory accounting for processes writing to 
GPFS
Sent by:gpfsug-discuss-boun...@spectrumscale.org




For any process with a large number of threads the VMM size has become an 
imaginary number ever since the glibc change to allocate a heap per thread.
I look to /proc/$pid/status to find the memory used by a proc  RSS + Swap + 
kernel page tables.

Jim

On Wednesday, March 6, 2019, 4:25:48 AM EST, Dorigo Alvise (PSI) 
 wrote:


Hello to everyone,
Here a PSI we're observing something that in principle seems strange (at least 
to me).
We run a Java application writing into disk by mean of a standard 
AsynchronousFileChannel, whose I do not the details.
There are two instances of this application: one runs on a node writing on a 
local drive, the other one runs writing on a GPFS mounted filesystem (this node 
is part of the cluster, no remote-mounting).

What we do see is that in the former the application has a lower sum VIRT+RES 
memory and the OS shows a really big cache usage; in the latter, OS's cache is 
negligible while VIRT+RES is very (even too) high (with VIRT very high).

So I wonder what is the difference... Writing into a GPFS mounted filesystem, 
as far as I understand, implies "talking" to the local mmfsd daemon which fills 
up its own pagepool... and then the system will asynchronously handle these 
pages to be written on real pdisk. But why the Linux kernel accounts so much 
memory to the process itself ? And why this large amount of memory is much more 
VIRT than RES ?

thanks in advance,

   Alvise
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss



___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss