Re: [gpfsug-discuss] Experience with CES NFS export management
On Tue, 24 Oct 2017 13:27:29 +0530, "Malahal R Naineni" said: > If you want to change multiple existing exports, you could use > undocumented option "--nfsnorestart" to mmnfs. This should add export > changes to NFS configuration but it won't restart nfs-ganesha service, so > you will not see immediate results of your changes in the running server. > Whenever you want your changes reflected, you could manually restart the > service using "mmces" command. I owe you a beverage of your choice if we ever are in the same place at the same time - the fact that Ganesha got restarted on all nodes at once thus preventing a rolling restart and avoiding service interruption was the single biggest Ganesha wart we've encountered. :) pgpsx2ycMuqFj.pgp Description: PGP signature ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Experience with CES NFS export management
As others have answered, 4.2.3 spectrum can add or remove exports without restarting nfs-ganesha service. Changing an existing export does need nfs-ganesha restart though. If you want to change multiple existing exports, you could use undocumented option "--nfsnorestart" to mmnfs. This should add export changes to NFS configuration but it won't restart nfs-ganesha service, so you will not see immediate results of your changes in the running server. Whenever you want your changes reflected, you could manually restart the service using "mmces" command. Regards, Malahal. From: Bryan Banister <bbanis...@jumptrading.com> To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org> Date: 10/23/2017 11:53 PM Subject: Re: [gpfsug-discuss] Experience with CES NFS export management Sent by:gpfsug-discuss-boun...@spectrumscale.org This becomes very disruptive when you have to add or remove many NFS exports. Is it possible to add and remove multiple entries at a time or is this YARFE time? -Bryan -Original Message- From: gpfsug-discuss-boun...@spectrumscale.org [ mailto:gpfsug-discuss-boun...@spectrumscale.org] On Behalf Of valdis.kletni...@vt.edu Sent: Monday, October 23, 2017 1:13 PM To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org> Subject: Re: [gpfsug-discuss] Experience with CES NFS export management On Mon, 23 Oct 2017 17:26:07 +0530, "Chetan R Kulkarni" said: > tests: > 1. created 1st nfs export - ganesha service was restarted > 2. created 4 more nfs exports (mmnfs export add path) > 3. changed 2 nfs exports (mmnfs export change path --nfschange); > 4. removed all 5 exports one by one (mmnfs export remove path) > 5. no nfs exports after step 4 on my test system. So, created a new nfs > export (which will be the 1st nfs export). > 6. change nfs export created in step 5 mmnfs export change --nfsadd seems to generate a restart as well. Particularly annoying when the currently running nfs.ganesha fails to stop rpc.statd on the way down, and then bringing it back up fails because the port is in use Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss=DwICAg=jf_iaSHvJObTbx-siA1ZOg=oaQVLOYto6Ftb8wAbynvIiIdh2UEjHxQByDz70-6a_0=dhIJJ5KI4U6ZUia7OPi_-AC3qBrYV9n93ww8Ffhl468=K4ii44lk1_auA_3g7SN-E1zmMZNtc1PqBSiQJVudc_w= ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Experience with CES NFS export management
we can not commit on timelines on mailing lists, but this is a known issue and will be addressed in a future release. sven On Mon, Oct 23, 2017, 11:23 AM Bryan Banister <bbanis...@jumptrading.com> wrote: > This becomes very disruptive when you have to add or remove many NFS > exports. Is it possible to add and remove multiple entries at a time or is > this YARFE time? > -Bryan > > -Original Message- > From: gpfsug-discuss-boun...@spectrumscale.org [mailto: > gpfsug-discuss-boun...@spectrumscale.org] On Behalf Of > valdis.kletni...@vt.edu > Sent: Monday, October 23, 2017 1:13 PM > To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org> > Subject: Re: [gpfsug-discuss] Experience with CES NFS export management > > On Mon, 23 Oct 2017 17:26:07 +0530, "Chetan R Kulkarni" said: > > > tests: > > 1. created 1st nfs export - ganesha service was restarted > > 2. created 4 more nfs exports (mmnfs export add path) > > 3. changed 2 nfs exports (mmnfs export change path --nfschange); > > 4. removed all 5 exports one by one (mmnfs export remove path) > > 5. no nfs exports after step 4 on my test system. So, created a new nfs > > export (which will be the 1st nfs export). > > 6. change nfs export created in step 5 > > mmnfs export change --nfsadd seems to generate a restart as well. > Particularly annoying when the currently running nfs.ganesha fails to > stop rpc.statd on the way down, and then bringing it back up fails because > the port is in use > > > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged information. > If you are not the intended recipient, you are hereby notified that any > review, dissemination or copying of this email is strictly prohibited, and > to please notify the sender immediately and destroy this email and any > attachments. Email transmission cannot be guaranteed to be secure or > error-free. The Company, therefore, does not make any guarantees as to the > completeness or accuracy of this email or any attachments. This email is > for informational purposes only and does not constitute a recommendation, > offer, request or solicitation of any kind to buy, sell, subscribe, redeem > or perform any type of transaction of a financial product. > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Experience with CES NFS export management
Hello Bryan, at least changing multiple entries at once is possible. You can copy /var/mmfs/ces/nfs-config/gpfs.ganesha.exports.conf to e.g. /tmp, modify the export (remove/add nodes or options) and load the changed config via "mmnfs export load " That way, only a single restart is issued for Ganesha on the CES nodes. Adding/removing I did not try so far, to be honest for use-cases this is rather static. Regards, Stefan - Original Message - > From: "Bryan Banister" <bbanis...@jumptrading.com> > To: "gpfsug main discussion list" <gpfsug-discuss@spectrumscale.org> > Sent: Monday, October 23, 2017 8:23:33 PM > Subject: Re: [gpfsug-discuss] Experience with CES NFS export management > This becomes very disruptive when you have to add or remove many NFS exports. > Is it possible to add and remove multiple entries at a time or is this YARFE > time? > -Bryan > > -Original Message- > From: gpfsug-discuss-boun...@spectrumscale.org > [mailto:gpfsug-discuss-boun...@spectrumscale.org] On Behalf Of > valdis.kletni...@vt.edu > Sent: Monday, October 23, 2017 1:13 PM > To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org> > Subject: Re: [gpfsug-discuss] Experience with CES NFS export management > > On Mon, 23 Oct 2017 17:26:07 +0530, "Chetan R Kulkarni" said: > >> tests: >> 1. created 1st nfs export - ganesha service was restarted >> 2. created 4 more nfs exports (mmnfs export add path) >> 3. changed 2 nfs exports (mmnfs export change path --nfschange); >> 4. removed all 5 exports one by one (mmnfs export remove path) >> 5. no nfs exports after step 4 on my test system. So, created a new nfs >> export (which will be the 1st nfs export). >> 6. change nfs export created in step 5 > > mmnfs export change --nfsadd seems to generate a restart as well. > Particularly annoying when the currently running nfs.ganesha fails to > stop rpc.statd on the way down, and then bringing it back up fails because > the port is in use > > > > Note: This email is for the confidential use of the named addressee(s) only > and > may contain proprietary, confidential or privileged information. If you are > not > the intended recipient, you are hereby notified that any review, dissemination > or copying of this email is strictly prohibited, and to please notify the > sender immediately and destroy this email and any attachments. Email > transmission cannot be guaranteed to be secure or error-free. The Company, > therefore, does not make any guarantees as to the completeness or accuracy of > this email or any attachments. This email is for informational purposes only > and does not constitute a recommendation, offer, request or solicitation of > any > kind to buy, sell, subscribe, redeem or perform any type of transaction of a > financial product. > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Experience with CES NFS export management
This becomes very disruptive when you have to add or remove many NFS exports. Is it possible to add and remove multiple entries at a time or is this YARFE time? -Bryan -Original Message- From: gpfsug-discuss-boun...@spectrumscale.org [mailto:gpfsug-discuss-boun...@spectrumscale.org] On Behalf Of valdis.kletni...@vt.edu Sent: Monday, October 23, 2017 1:13 PM To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org> Subject: Re: [gpfsug-discuss] Experience with CES NFS export management On Mon, 23 Oct 2017 17:26:07 +0530, "Chetan R Kulkarni" said: > tests: > 1. created 1st nfs export - ganesha service was restarted > 2. created 4 more nfs exports (mmnfs export add path) > 3. changed 2 nfs exports (mmnfs export change path --nfschange); > 4. removed all 5 exports one by one (mmnfs export remove path) > 5. no nfs exports after step 4 on my test system. So, created a new nfs > export (which will be the 1st nfs export). > 6. change nfs export created in step 5 mmnfs export change --nfsadd seems to generate a restart as well. Particularly annoying when the currently running nfs.ganesha fails to stop rpc.statd on the way down, and then bringing it back up fails because the port is in use Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Experience with CES NFS export management
On Mon, 23 Oct 2017 17:26:07 +0530, "Chetan R Kulkarni" said: > tests: > 1. created 1st nfs export - ganesha service was restarted > 2. created 4 more nfs exports (mmnfs export add path) > 3. changed 2 nfs exports (mmnfs export change path --nfschange); > 4. removed all 5 exports one by one (mmnfs export remove path) > 5. no nfs exports after step 4 on my test system. So, created a new nfs > export (which will be the 1st nfs export). > 6. change nfs export created in step 5 mmnfs export change --nfsadd seems to generate a restart as well. Particularly annoying when the currently running nfs.ganesha fails to stop rpc.statd on the way down, and then bringing it back up fails because the port is in use pgpV4o_ofL95x.pgp Description: PGP signature ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Experience with CES NFS export management
Dear Chetan, interesting. I’m running ISS 4.2.3-4 and it seems to ship with nfs-ganesha-2.3.2. So are you already using a future ISS version? Here is what I see: [root@datdsst102 pr74cu-dss-0002]# mmnfs export list Path Delegations Clients -- /dss/dsstestfs01/pr74cu-dss-0002 NONE10.156.29.73 /dss/dsstestfs01/pr74cu-dss-0002 NONE10.156.29.72 [root@datdsst102 pr74cu-dss-0002]# mmnfs export change /dss/dsstestfs01/pr74cu-dss-0002 --nfschange "10.156.29.72(access_type=RW,squash=no_root_squash,protocols=4,transports=tcp,sectype=sys,manage_gids=true)" datdsst102.dss.lrz.de<http://datdsst102.dss.lrz.de>: Redirecting to /bin/systemctl stop nfs-ganesha.service datdsst102.dss.lrz.de<http://datdsst102.dss.lrz.de>: Redirecting to /bin/systemctl start nfs-ganesha.service NFS Configuration successfully changed. NFS server restarted on all NFS nodes on which NFS server is running. [root@datdsst102 pr74cu-dss-0002]# mmnfs export change /dss/dsstestfs01/pr74cu-dss-0002 --nfschange "10.156.29.72(access_type=RW,squash=no_root_squash,protocols=4,transports=tcp,sectype=sys,manage_gids=true)" datdsst102.dss.lrz.de<http://datdsst102.dss.lrz.de>: Redirecting to /bin/systemctl stop nfs-ganesha.service datdsst102.dss.lrz.de<http://datdsst102.dss.lrz.de>: Redirecting to /bin/systemctl start nfs-ganesha.service NFS Configuration successfully changed. NFS server restarted on all NFS nodes on which NFS server is running. [root@datdsst102 pr74cu-dss-0002]# mmnfs export change /dss/dsstestfs01/pr74cu-dss-0002 --nfsadd "10.156.29.74(access_type=RW,squash=no_root_squash,protocols=4,transports=tcp,sectype=sys,manage_gids=true)" datdsst102.dss.lrz.de<http://datdsst102.dss.lrz.de>: Redirecting to /bin/systemctl stop nfs-ganesha.service datdsst102.dss.lrz.de<http://datdsst102.dss.lrz.de>: Redirecting to /bin/systemctl start nfs-ganesha.service NFS Configuration successfully changed. NFS server restarted on all NFS nodes on which NFS server is running. [root@datdsst102 ~]# mmnfs export change /dss/dsstestfs01/pr74cu-dss-0002 --nfsremove 10.156.29.74 datdsst102.dss.lrz.de<http://datdsst102.dss.lrz.de>: Redirecting to /bin/systemctl stop nfs-ganesha.service datdsst102.dss.lrz.de<http://datdsst102.dss.lrz.de>: Redirecting to /bin/systemctl start nfs-ganesha.service NFS Configuration successfully changed. NFS server restarted on all NFS nodes on which NFS server is running. Best Regards, Stephan Peinkofer -- Stephan Peinkofer Dipl. Inf. (FH), M. Sc. (TUM) Leibniz Supercomputing Centre Data and Storage Division Boltzmannstraße 1, 85748 Garching b. München Tel: +49(0)89 35831-8715 Fax: +49(0)89 35831-9700 URL: http://www.lrz.de On 23. Oct 2017, at 13:56, Chetan R Kulkarni <chetk...@in.ibm.com<mailto:chetk...@in.ibm.com>> wrote: Hi Stephan, I observed ganesha service getting restarted only after adding first nfs export. For rest of the operations (e.g. adding more nfs exports, changing nfs exports, removing nfs exports); ganesha service doesn't restart. My observations are based on following simple tests. I ran them against rhel7.3 test cluster having nfs-ganesha-2.5.2. tests: 1. created 1st nfs export - ganesha service was restarted 2. created 4 more nfs exports (mmnfs export add path) 3. changed 2 nfs exports (mmnfs export change path --nfschange); 4. removed all 5 exports one by one (mmnfs export remove path) 5. no nfs exports after step 4 on my test system. So, created a new nfs export (which will be the 1st nfs export). 6. change nfs export created in step 5 results observed: ganesha service restarted for test 1 and test 5. For rest tests (2,3,4,6); ganesha service didn't restart. Thanks, Chetan. "Peinkofer, Stephan" ---10/23/2017 04:11:33 PM---Dear List, I’m currently working on a self service portal for managing NFS exports of ISS. Basically From: "Peinkofer, Stephan" <stephan.peinko...@lrz.de<mailto:stephan.peinko...@lrz.de>> To: "gpfsug-discuss@spectrumscale.org<mailto:gpfsug-discuss@spectrumscale.org>" <gpfsug-discuss@spectrumscale.org<mailto:gpfsug-discuss@spectrumscale.org>> Date: 10/23/2017 04:11 PM Subject: [gpfsug-discuss] Experience with CES NFS export management Sent by: gpfsug-discuss-boun...@spectrumscale.org<mailto:gpfsug-discuss-boun...@spectrumscale.org> Dear List, I’m currently working on a self service portal for managing NFS exports of ISS. Basically something very similar to OpenStack Manila but tailored to our specific needs. While it was very easy to do this using the great REST API of ISS, I stumbled across a fact that may be even a show stopper: According to the documentation for mmnfs, each time we create/change/delete a NFS export via mmnfs, ganesha service is restar
Re: [gpfsug-discuss] Experience with CES NFS export management
Hi Stephan, I observed ganesha service getting restarted only after adding first nfs export. For rest of the operations (e.g. adding more nfs exports, changing nfs exports, removing nfs exports); ganesha service doesn't restart. My observations are based on following simple tests. I ran them against rhel7.3 test cluster having nfs-ganesha-2.5.2. tests: 1. created 1st nfs export - ganesha service was restarted 2. created 4 more nfs exports (mmnfs export add path) 3. changed 2 nfs exports (mmnfs export change path --nfschange); 4. removed all 5 exports one by one (mmnfs export remove path) 5. no nfs exports after step 4 on my test system. So, created a new nfs export (which will be the 1st nfs export). 6. change nfs export created in step 5 results observed: ganesha service restarted for test 1 and test 5. For rest tests (2,3,4,6); ganesha service didn't restart. Thanks, Chetan. From: "Peinkofer, Stephan" <stephan.peinko...@lrz.de> To: "gpfsug-discuss@spectrumscale.org" <gpfsug-discuss@spectrumscale.org> Date: 10/23/2017 04:11 PM Subject: [gpfsug-discuss] Experience with CES NFS export management Sent by:gpfsug-discuss-boun...@spectrumscale.org Dear List, I’m currently working on a self service portal for managing NFS exports of ISS. Basically something very similar to OpenStack Manila but tailored to our specific needs. While it was very easy to do this using the great REST API of ISS, I stumbled across a fact that may be even a show stopper: According to the documentation for mmnfs, each time we create/change/delete a NFS export via mmnfs, ganesha service is restarted on all nodes. I assume that this behaviour may cause problems (at least IO stalls) on clients mounted the filesystem. So my question is, what is your experience with CES NFS export management. Do you see any problems when you add/change/delete exports and ganesha gets restarted? Are there any (supported) workarounds for this problem? PS: As I think in 2017 CES Exports should be manageable without service disruptions (and ganesha provides facilities to do so), I filed an RFE for this: https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe_ID=111918 Many thanks in advance. Best Regards, Stephan Peinkofer -- Stephan Peinkofer Dipl. Inf. (FH), M. Sc. (TUM) Leibniz Supercomputing Centre Data and Storage Division Boltzmannstraße 1, 85748 Garching b. München Tel: +49(0)89 35831-8715 Fax: +49(0)89 35831-9700 URL: http://www.lrz.de ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss=DwICAg=jf_iaSHvJObTbx-siA1ZOg=uic-29lyJ5TCiTRi0FyznYhKJx5I7Vzu80WyYuZ4_iM=ghcZYswqgF3beYOogGGLsT1RyDRZrbLXdzp3Fbjmfrg=TUm7BM3sY75Nc20gOfhz9lvDgYJse0TM6-tIW8I1QiI= ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Experience with CES NFS export management
You can lower LEASE_LIFETIME and GRACE_PERIOD to shorten the time it's in grace, to make it more bearable. Making export changes dynamic is something that's fixed in newer versions of nfs-ganesha than what's shipped with Scale: https://github.com/nfs-ganesha/nfs-ganesha/releases/tag/V2.4.0: "dynamic EXPORT configuration update (via dBus and SIGHUP)" Hopefully someone can comment on when we'll see nfs-ganesha v2.4+ included with Scale. -jf On Mon, Oct 23, 2017 at 12:41 PM, Peinkofer, Stephan < stephan.peinko...@lrz.de> wrote: > Dear List, > > I’m currently working on a self service portal for managing NFS exports of > ISS. Basically something very similar to OpenStack Manila but tailored to > our specific needs. > While it was very easy to do this using the great REST API of ISS, I > stumbled across a fact that may be even a show stopper: According to the > documentation for mmnfs, each time we > create/change/delete a NFS export via mmnfs, ganesha service is restarted > on all nodes. > > I assume that this behaviour may cause problems (at least IO stalls) on > clients mounted the filesystem. So my question is, what is your experience > with CES NFS export management. > Do you see any problems when you add/change/delete exports and ganesha > gets restarted? > > Are there any (supported) workarounds for this problem? > > PS: As I think in 2017 CES Exports should be manageable without service > disruptions (and ganesha provides facilities to do so), I filed an RFE for > this: https://www.ibm.com/developerworks/rfe/execute? > use_case=viewRfe_ID=111918 > > Many thanks in advance. > Best Regards, > Stephan Peinkofer > -- > Stephan Peinkofer > Dipl. Inf. (FH), M. Sc. (TUM) > > Leibniz Supercomputing Centre > Data and Storage Division > Boltzmannstraße 1, 85748 Garching b. München > Tel: +49(0)89 35831-8715 <+49%2089%20358318715> Fax: +49(0)89 > 35831-9700 <+49%2089%20358319700> > URL: http://www.lrz.de > > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
