Re: [Nfs-ganesha-devel] Problems in /usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling exports from gluster
I see that NFS-Ganesha is no longer running after the unexport. Can you move to
the latest sources which has Soumya's fix and try once?
Even if the config file is still included in the ganesha.conf, dynamically
removing that export should not affect the process itself.
Meghana
- Original Message -
From: Alessandro De Salvo alessandro.desa...@roma1.infn.it
To: Meghana Madhusudhan mmadh...@redhat.com
Cc: gluster-us...@gluster.org, nfs-ganesha-devel@lists.sourceforge.net, Soumya
Koduri skod...@redhat.com
Sent: Thursday, June 18, 2015 7:55:30 PM
Subject: Re: [Nfs-ganesha-devel] Problems in /usr/libexec/ganesha/dbus-send.sh
and ganesha dbus interface when disabling exports from gluster
Hi Meghana,
Il giorno 18/giu/2015, alle ore 16:06, Meghana Madhusudhan
mmadh...@redhat.com ha scritto:
Hi Allesandro,
I need the following output from you,
1. After you execute ganesha.enable on command.
ps aux | grep ganesha
root 6699 94.0 1.1 953488 193272 ? Ssl Jun17 1137:36
/usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
NIV_EVENT -p /var/run/ganesha.nfsd.pid
root 22263 0.0 0.0 112644 960 pts/0S+ 16:14 0:00 grep
--color=auto ganesha
(both nodes of the cluster show the same, apart from the PIDs, of course)
showmount -e localhost
Export list for node1:
/atlas-backup-01 (everyone)
The same output is shown on the other node.
cat /etc/ganesha/ganesha.conf
# Ganesha options file
%include /etc/ganesha/ganesha-opts.conf
%include /etc/ganesha/exports/export.atlas-backup-01.conf
The first include just point to a different file with the following lines, to
change the rquota port
NFS_Core_Param {
#Use a non-privileged port for RQuota
Rquota_Port = 4501;
}
cat /etc/ganesha/exports/export.VOLNAME.conf or
/usr//export.VOLNAME.conf”
# WARNING : Using Gluster CLI will overwrite manual
# changes made to this file. To avoid it, edit the
# file, copy it over to all the NFS-Ganesha nodes
# and run ganesha-ha.sh --refresh-config.
EXPORT{
Export_Id= 2 ;
Path = /atlas-backup-01;
FSAL {
name = GLUSTER;
hostname=localhost;
volume=atlas-backup-01;
}
Access_type = RW;
Disable_ACL = true;
Squash=No_root_squash;
Pseudo=/atlas-backup-01;
Protocols = 3, 4 ;
Transports = UDP,TCP;
SecType = sys;
}
tail /var/log/ganesha.log or wherever the ganesha log is.
17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main]
nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main]
nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main]
nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[reaper]
nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE
17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main]
nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
17/06/2015 20:01:01 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main]
nfs_start :NFS STARTUP :EVENT :-
17/06/2015 20:01:01 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main]
nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED
17/06/2015 20:01:01 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main]
nfs_start :NFS STARTUP :EVENT :-
17/06/2015 20:02:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[reaper]
nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE
18/06/2015 16:13:50 : epoch 5581b5da : node2 :
ganesha.nfsd-26983[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume
atlas-backup-01 exported at : '/'
2. gluster vol set VOLNAME ganesha.enable off
Output of the same as above.
# gluster vol set atlas-backup-01 ganesha.enable off
volume set: failed: Dynamic export addition/deletion failed. Please see log
file for details
# ps aux | grep ganesha
root 32033 0.0 0.0 112640 960 pts/0S+ 16:20 0:00 grep
--color=auto ganesha
[This is due to the fact that the export file was deleted by the config still
references it, so ganesha fails to resume]
# cat /etc/ganesha/ganesha.conf
# Ganesha options file
%include /etc/ganesha/ganesha-opts.conf
%include /etc/ganesha/exports/export.atlas-backup-01.conf
# cat /etc/ganesha/exports/export.atlas-backup-01.conf
cat: /etc/ganesha/exports/export.atlas-backup-01.conf: No such file or directory
# tail /var/log/ganesha.log
18/06/2015 16:19:20 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main]
glusterfs_create_export :FSAL :EVENT :Volume atlas-backup-01 exported at : '/'
18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main]
lower_my_caps :NFS STARTUP :EVENT
Re: [Nfs-ganesha-devel] Problems in /usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling exports from gluster
Hi Meghana,
indeed, as I just reported to Soumya and Malahal too, the new version is
indeed solving the problem.
Many thanks to everyone for solving it!
Alessandro
On Fri, 2015-06-19 at 03:28 -0400, Meghana Madhusudhan wrote:
I see that NFS-Ganesha is no longer running after the unexport. Can you move
to the latest sources which has Soumya's fix and try once?
Even if the config file is still included in the ganesha.conf, dynamically
removing that export should not affect the process itself.
Meghana
- Original Message -
From: Alessandro De Salvo alessandro.desa...@roma1.infn.it
To: Meghana Madhusudhan mmadh...@redhat.com
Cc: gluster-us...@gluster.org, nfs-ganesha-devel@lists.sourceforge.net,
Soumya Koduri skod...@redhat.com
Sent: Thursday, June 18, 2015 7:55:30 PM
Subject: Re: [Nfs-ganesha-devel] Problems in
/usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling
exports from gluster
Hi Meghana,
Il giorno 18/giu/2015, alle ore 16:06, Meghana Madhusudhan
mmadh...@redhat.com ha scritto:
Hi Allesandro,
I need the following output from you,
1. After you execute ganesha.enable on command.
ps aux | grep ganesha
root 6699 94.0 1.1 953488 193272 ? Ssl Jun17 1137:36
/usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
NIV_EVENT -p /var/run/ganesha.nfsd.pid
root 22263 0.0 0.0 112644 960 pts/0S+ 16:14 0:00 grep
--color=auto ganesha
(both nodes of the cluster show the same, apart from the PIDs, of course)
showmount -e localhost
Export list for node1:
/atlas-backup-01 (everyone)
The same output is shown on the other node.
cat /etc/ganesha/ganesha.conf
# Ganesha options file
%include /etc/ganesha/ganesha-opts.conf
%include /etc/ganesha/exports/export.atlas-backup-01.conf
The first include just point to a different file with the following lines, to
change the rquota port
NFS_Core_Param {
#Use a non-privileged port for RQuota
Rquota_Port = 4501;
}
cat /etc/ganesha/exports/export.VOLNAME.conf or
/usr//export.VOLNAME.conf”
# WARNING : Using Gluster CLI will overwrite manual
# changes made to this file. To avoid it, edit the
# file, copy it over to all the NFS-Ganesha nodes
# and run ganesha-ha.sh --refresh-config.
EXPORT{
Export_Id= 2 ;
Path = /atlas-backup-01;
FSAL {
name = GLUSTER;
hostname=localhost;
volume=atlas-backup-01;
}
Access_type = RW;
Disable_ACL = true;
Squash=No_root_squash;
Pseudo=/atlas-backup-01;
Protocols = 3, 4 ;
Transports = UDP,TCP;
SecType = sys;
}
tail /var/log/ganesha.log or wherever the ganesha log is.
17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main]
nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main]
nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main]
nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[reaper]
nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE
17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main]
nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
17/06/2015 20:01:01 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main]
nfs_start :NFS STARTUP :EVENT
:-
17/06/2015 20:01:01 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main]
nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED
17/06/2015 20:01:01 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main]
nfs_start :NFS STARTUP :EVENT
:-
17/06/2015 20:02:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[reaper]
nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE
18/06/2015 16:13:50 : epoch 5581b5da : node2 :
ganesha.nfsd-26983[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT
:Volume atlas-backup-01 exported at : '/'
2. gluster vol set VOLNAME ganesha.enable off
Output of the same as above.
# gluster vol set atlas-backup-01 ganesha.enable off
volume set: failed: Dynamic export addition/deletion failed. Please see log
file for details
# ps aux | grep ganesha
root 32033 0.0 0.0 112640 960 pts/0S+ 16:20 0:00 grep
--color=auto ganesha
[This is due to the fact that the export file was deleted by the config still
references it, so ganesha fails to resume]
# cat /etc/ganesha/ganesha.conf
# Ganesha options file
%include /etc/ganesha/ganesha-opts.conf
%include /etc/ganesha/exports/export.atlas-backup-01.conf
# cat /etc/ganesha/exports/export.atlas-backup-01.conf
cat
Re: [Nfs-ganesha-devel] Problems in /usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling exports from gluster
Hi Allesandro, I need the following output from you, 1. After you execute ganesha.enable on command. ps aux | grep ganesha showmount -e localhost cat /etc/ganesha/ganesha.conf cat /etc/ganesha/exports/export.VOLNAME.conf or /usr//export.VOLNAME.conf tail /var/log/ganesha.log or wherever the ganesha log is. 2. gluster vol set VOLNAME ganesha.enable off Output of the same as above. Thanks for your perseverance :) Meghana - Original Message - From: Alessandro De Salvo alessandro.desa...@roma1.infn.it To: Meghana Madhusudhan mmadh...@redhat.com Cc: gluster-us...@gluster.org, nfs-ganesha-devel@lists.sourceforge.net, Soumya Koduri skod...@redhat.com Sent: Thursday, June 18, 2015 7:24:55 PM Subject: Re: [Nfs-ganesha-devel] Problems in /usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling exports from gluster Hi Meghana, Il giorno 18/giu/2015, alle ore 07:04, Meghana Madhusudhan mmadh...@redhat.com ha scritto: On 06/17/2015 10:57 PM, Alessandro De Salvo wrote: Hi, when disabling exports from gluster 3.7.1, by using gluster vol set volume ganesha.enable off, I always get the following error: Error: Dynamic export addition/deletion failed. Please see log file for details This message is produced by the failure of /usr/libexec/ganesha/dbus-send.sh, and in fact if I manually perform the command to remove the share I see: you got it wrong. '/usr/libexec/ganesha/dbus-send.sh' is used by Gluster-CLI to unexport the volume gluster volume set volname ganesha.enable off which rightly deletes the export file too while un-exporting the volume. # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2 Error org.freedesktop.DBus.Error.NoReply: Message did not receive a reply (timeout by message bus) So, there is a timeout and it fails completely. Check if nfs-ganesha is still running. There was a bug in unexporting the volume. Its been fixed recently in V2.3-dev, yet to be back-ported to V2.2-stable branch. https://review.gerrithub.io/#/c/236129/ Thanks, Soumya In this case I think there is a bug in /usr/libexec/ganesha/dbus-send.sh, since it blindly deletes the share config if the RemoveExport fails (function check_cmd_status()), but leaves the %include inside ganesha.conf as in the check_cmd_status() there is a runaway condition and the other removal statements are then not executed. I believe the logic should be fixed here, otherwise even a restart of the service will fail due to the bad configuration. Yes. I see that the sed -i /$VOL.conf/d $CONF is placed after the check_cmd_status. I shall send a fix upstream in a related bug. But dynamic export removal will fail in three cases, 1. nfs-ganesha is not running. no, it was running 2. The export file that is particular to that volume is somehow deleted before you perform the removal. It does depend on that file to get the export ID. I tried to comment out the rm in check_cmd_status to avoid this race condition, but it did not solve the problems. 3. The bug that Soumya pointed out. This might well be the real cause! If it is failing consistently, there could be something that you are missing. If you can send the exact sequence of sequence of steps that you have executed, I can help you with it. Yes, it’s failing consistently, unless as I said I do a DisplayExport before the RemoveExport, in which case it always works. Ideally after exporting a particular volume, you'll see an entry in the /etc/ganesha/ganesha.conf file and the export file in /etc/ganesha/exports dir. And this works perfectly, I see them correctly. If you have this in place and nfs-ganesha running, then dynamic export removal should work just fine. But this is not, at least in my case. The command I’m using are just the following: gluster vol set volume ganesha.enable on gluster vol set volume ganesha.enable off It normally wait a few seconds between the two commands, to give time to ganesha to actually export the volume. The export is always failing as described, unless I add the DisplayExport in dbus-send.sh before RemoveExport. Many thanks for the help, Alessandro Meghana What’s more worrying is the problem with the dbus. Issuing a DisplayExport before the RemoveExport apparently fixes the problem, so something like this always works: # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.DisplayExport uint16:2 # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2 So, it’s like the DisplayExport is forcing someway a refresh that is needed by the RemoveExport. Any idea why? I’m using the latest version of ganesha 2.2.0, i.e. 2.2.0-3. Thanks, Alessandro PS: sorry for reporting
Re: [Nfs-ganesha-devel] Problems in /usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling exports from gluster
Hi Meghana, Il giorno 18/giu/2015, alle ore 07:04, Meghana Madhusudhan mmadh...@redhat.com ha scritto: On 06/17/2015 10:57 PM, Alessandro De Salvo wrote: Hi, when disabling exports from gluster 3.7.1, by using gluster vol set volume ganesha.enable off, I always get the following error: Error: Dynamic export addition/deletion failed. Please see log file for details This message is produced by the failure of /usr/libexec/ganesha/dbus-send.sh, and in fact if I manually perform the command to remove the share I see: you got it wrong. '/usr/libexec/ganesha/dbus-send.sh' is used by Gluster-CLI to unexport the volume gluster volume set volname ganesha.enable off which rightly deletes the export file too while un-exporting the volume. # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2 Error org.freedesktop.DBus.Error.NoReply: Message did not receive a reply (timeout by message bus) So, there is a timeout and it fails completely. Check if nfs-ganesha is still running. There was a bug in unexporting the volume. Its been fixed recently in V2.3-dev, yet to be back-ported to V2.2-stable branch. https://review.gerrithub.io/#/c/236129/ Thanks, Soumya In this case I think there is a bug in /usr/libexec/ganesha/dbus-send.sh, since it blindly deletes the share config if the RemoveExport fails (function check_cmd_status()), but leaves the %include inside ganesha.conf as in the check_cmd_status() there is a runaway condition and the other removal statements are then not executed. I believe the logic should be fixed here, otherwise even a restart of the service will fail due to the bad configuration. Yes. I see that the sed -i /$VOL.conf/d $CONF is placed after the check_cmd_status. I shall send a fix upstream in a related bug. But dynamic export removal will fail in three cases, 1. nfs-ganesha is not running. no, it was running 2. The export file that is particular to that volume is somehow deleted before you perform the removal. It does depend on that file to get the export ID. I tried to comment out the rm in check_cmd_status to avoid this race condition, but it did not solve the problems. 3. The bug that Soumya pointed out. This might well be the real cause! If it is failing consistently, there could be something that you are missing. If you can send the exact sequence of sequence of steps that you have executed, I can help you with it. Yes, it’s failing consistently, unless as I said I do a DisplayExport before the RemoveExport, in which case it always works. Ideally after exporting a particular volume, you'll see an entry in the /etc/ganesha/ganesha.conf file and the export file in /etc/ganesha/exports dir. And this works perfectly, I see them correctly. If you have this in place and nfs-ganesha running, then dynamic export removal should work just fine. But this is not, at least in my case. The command I’m using are just the following: gluster vol set volume ganesha.enable on gluster vol set volume ganesha.enable off It normally wait a few seconds between the two commands, to give time to ganesha to actually export the volume. The export is always failing as described, unless I add the DisplayExport in dbus-send.sh before RemoveExport. Many thanks for the help, Alessandro Meghana What’s more worrying is the problem with the dbus. Issuing a DisplayExport before the RemoveExport apparently fixes the problem, so something like this always works: # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.DisplayExport uint16:2 # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2 So, it’s like the DisplayExport is forcing someway a refresh that is needed by the RemoveExport. Any idea why? I’m using the latest version of ganesha 2.2.0, i.e. 2.2.0-3. Thanks, Alessandro PS: sorry for reporting so many things in a few days :-) -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel smime.p7s Description: S/MIME cryptographic signature -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net
Re: [Nfs-ganesha-devel] Problems in /usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling exports from gluster
...@redhat.com Sent: Thursday, June 18, 2015 7:24:55 PM Subject: Re: [Nfs-ganesha-devel] Problems in /usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling exports from gluster Hi Meghana, Il giorno 18/giu/2015, alle ore 07:04, Meghana Madhusudhan mmadh...@redhat.com ha scritto: On 06/17/2015 10:57 PM, Alessandro De Salvo wrote: Hi, when disabling exports from gluster 3.7.1, by using gluster vol set volume ganesha.enable off, I always get the following error: Error: Dynamic export addition/deletion failed. Please see log file for details This message is produced by the failure of /usr/libexec/ganesha/dbus-send.sh, and in fact if I manually perform the command to remove the share I see: you got it wrong. '/usr/libexec/ganesha/dbus-send.sh' is used by Gluster-CLI to unexport the volume gluster volume set volname ganesha.enable off which rightly deletes the export file too while un-exporting the volume. # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2 Error org.freedesktop.DBus.Error.NoReply: Message did not receive a reply (timeout by message bus) So, there is a timeout and it fails completely. Check if nfs-ganesha is still running. There was a bug in unexporting the volume. Its been fixed recently in V2.3-dev, yet to be back-ported to V2.2-stable branch. https://review.gerrithub.io/#/c/236129/ Thanks, Soumya In this case I think there is a bug in /usr/libexec/ganesha/dbus-send.sh, since it blindly deletes the share config if the RemoveExport fails (function check_cmd_status()), but leaves the %include inside ganesha.conf as in the check_cmd_status() there is a runaway condition and the other removal statements are then not executed. I believe the logic should be fixed here, otherwise even a restart of the service will fail due to the bad configuration. Yes. I see that the sed -i /$VOL.conf/d $CONF is placed after the check_cmd_status. I shall send a fix upstream in a related bug. But dynamic export removal will fail in three cases, 1. nfs-ganesha is not running. no, it was running 2. The export file that is particular to that volume is somehow deleted before you perform the removal. It does depend on that file to get the export ID. I tried to comment out the rm in check_cmd_status to avoid this race condition, but it did not solve the problems. 3. The bug that Soumya pointed out. This might well be the real cause! If it is failing consistently, there could be something that you are missing. If you can send the exact sequence of sequence of steps that you have executed, I can help you with it. Yes, it’s failing consistently, unless as I said I do a DisplayExport before the RemoveExport, in which case it always works. Ideally after exporting a particular volume, you'll see an entry in the /etc/ganesha/ganesha.conf file and the export file in /etc/ganesha/exports dir. And this works perfectly, I see them correctly. If you have this in place and nfs-ganesha running, then dynamic export removal should work just fine. But this is not, at least in my case. The command I’m using are just the following: gluster vol set volume ganesha.enable on gluster vol set volume ganesha.enable off It normally wait a few seconds between the two commands, to give time to ganesha to actually export the volume. The export is always failing as described, unless I add the DisplayExport in dbus-send.sh before RemoveExport. Many thanks for the help, Alessandro Meghana What’s more worrying is the problem with the dbus. Issuing a DisplayExport before the RemoveExport apparently fixes the problem, so something like this always works: # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.DisplayExport uint16:2 # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2 So, it’s like the DisplayExport is forcing someway a refresh that is needed by the RemoveExport. Any idea why? I’m using the latest version of ganesha 2.2.0, i.e. 2.2.0-3. Thanks, Alessandro PS: sorry for reporting so many things in a few days :-) -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel smime.p7s Description: S/MIME
Re: [Nfs-ganesha-devel] Problems in /usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling exports from gluster
I have not enabled the FULL_DEBUG but if you need it I can do it. Thanks for your perseverance :) I have to thank you for the help! :-) Cheers, Alessandro Meghana - Original Message - From: Alessandro De Salvo alessandro.desa...@roma1.infn.it To: Meghana Madhusudhan mmadh...@redhat.com Cc: gluster-us...@gluster.org, nfs-ganesha-devel@lists.sourceforge.net, Soumya Koduri skod...@redhat.com Sent: Thursday, June 18, 2015 7:24:55 PM Subject: Re: [Nfs-ganesha-devel] Problems in /usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling exports from gluster Hi Meghana, Il giorno 18/giu/2015, alle ore 07:04, Meghana Madhusudhan mmadh...@redhat.com ha scritto: On 06/17/2015 10:57 PM, Alessandro De Salvo wrote: Hi, when disabling exports from gluster 3.7.1, by using gluster vol set volume ganesha.enable off, I always get the following error: Error: Dynamic export addition/deletion failed. Please see log file for details This message is produced by the failure of /usr/libexec/ganesha/dbus-send.sh, and in fact if I manually perform the command to remove the share I see: you got it wrong. '/usr/libexec/ganesha/dbus-send.sh' is used by Gluster-CLI to unexport the volume gluster volume set volname ganesha.enable off which rightly deletes the export file too while un-exporting the volume. # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2 Error org.freedesktop.DBus.Error.NoReply: Message did not receive a reply (timeout by message bus) So, there is a timeout and it fails completely. Check if nfs-ganesha is still running. There was a bug in unexporting the volume. Its been fixed recently in V2.3-dev, yet to be back-ported to V2.2-stable branch. https://review.gerrithub.io/#/c/236129/ Thanks, Soumya In this case I think there is a bug in /usr/libexec/ganesha/dbus-send.sh, since it blindly deletes the share config if the RemoveExport fails (function check_cmd_status()), but leaves the %include inside ganesha.conf as in the check_cmd_status() there is a runaway condition and the other removal statements are then not executed. I believe the logic should be fixed here, otherwise even a restart of the service will fail due to the bad configuration. Yes. I see that the sed -i /$VOL.conf/d $CONF is placed after the check_cmd_status. I shall send a fix upstream in a related bug. But dynamic export removal will fail in three cases, 1. nfs-ganesha is not running. no, it was running 2. The export file that is particular to that volume is somehow deleted before you perform the removal. It does depend on that file to get the export ID. I tried to comment out the rm in check_cmd_status to avoid this race condition, but it did not solve the problems. 3. The bug that Soumya pointed out. This might well be the real cause! If it is failing consistently, there could be something that you are missing. If you can send the exact sequence of sequence of steps that you have executed, I can help you with it. Yes, it’s failing consistently, unless as I said I do a DisplayExport before the RemoveExport, in which case it always works. Ideally after exporting a particular volume, you'll see an entry in the /etc/ganesha/ganesha.conf file and the export file in /etc/ganesha/exports dir. And this works perfectly, I see them correctly. If you have this in place and nfs-ganesha running, then dynamic export removal should work just fine. But this is not, at least in my case. The command I’m using are just the following: gluster vol set volume ganesha.enable on gluster vol set volume ganesha.enable off It normally wait a few seconds between the two commands, to give time to ganesha to actually export the volume. The export is always failing as described, unless I add the DisplayExport in dbus-send.sh before RemoveExport. Many thanks for the help, Alessandro Meghana What’s more worrying is the problem with the dbus. Issuing a DisplayExport before the RemoveExport apparently fixes the problem, so something like this always works: # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.DisplayExport uint16:2 # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2 So, it’s like the DisplayExport is forcing someway a refresh that is needed by the RemoveExport. Any idea why? I’m using the latest version of ganesha 2.2.0, i.e. 2.2.0-3. Thanks, Alessandro PS: sorry for reporting so many things in a few days :-) -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https
Re: [Nfs-ganesha-devel] Problems in /usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling exports from gluster
On 06/18/2015 07:39 PM, Malahal Naineni wrote: I still have not looked at the log messages, but I see the dbus thread waiting for the upcall thread to complete when an export is removed. Is there is a time limit on how the upcall thread gets blocked? A variable called 'destroy_mode' is used to achieve that. This variable is set to true during unexport. Upcall thread checks this value in a loop before polling for events and exits when set to true. In GPFS fsal, we actually send a command to return the upcall thread with (THREAD_STOP) and then wait for the upcall thread. Thanks for the pointer. We plan to optimize our upcall mechanism by using callbacks/cond_variables instead of polling in the near future. We shall check on reducing the number of threads too dedicated for it. Thanks, Soumya Regards, Malahal. PS: GPFS has only one upcall thread for the entire file system image (not for each export). Alessandro De Salvo [alessandro.desa...@roma1.infn.it] wrote: Hi Malahal, Il giorno 17/giu/2015, alle ore 19:51, Malahal Naineni mala...@us.ibm.com ha scritto: Alessandro De Salvo [alessandro.desa...@roma1.infn.it] wrote: What’s more worrying is the problem with the dbus. Issuing a DisplayExport before the RemoveExport apparently fixes the problem, so something like this always works: # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.DisplayExport uint16:2 # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2 So, it’s like the DisplayExport is forcing someway a refresh that is needed by the RemoveExport. Any idea why? I’m using the latest version of ganesha 2.2.0, i.e. 2.2.0-3. I used the same exact command above (the second one that removes an export) after restarting ganesha, and it just worked fine. I use GPFS FSAL (neither gluster nor VFS). Not sure why you need to use DisplayExport before using RemoveExport. Try to trace 'DBUS' component at FULL_DEBUG (or maybe everything at FULL_DEBUG) and post the log. The error you reported means we are NOT responding to the dbus message which is very odd! Indeed! This is why I was worried :-) I’m attaching the ganesha.log with NIV_FULL_DEBUG, you can see the restart at the end, due to my attempt of RemoveExport, which always gave me the same error as before. Thanks, Alessandro Reagrds, Malahal. -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Problems in /usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling exports from gluster
Hi Soumya,
Il giorno 17/giu/2015, alle ore 19:41, Soumya Koduri skod...@redhat.com ha
scritto:
On 06/17/2015 10:57 PM, Alessandro De Salvo wrote:
Hi,
when disabling exports from gluster 3.7.1, by using gluster vol set volume
ganesha.enable off, I always get the following error:
Error: Dynamic export addition/deletion failed. Please see log file for
details
This message is produced by the failure of
/usr/libexec/ganesha/dbus-send.sh, and in fact if I manually perform the
command to remove the share I see:
you got it wrong. '/usr/libexec/ganesha/dbus-send.sh' is used by Gluster-CLI
to unexport the volume gluster volume set volname ganesha.enable off
which rightly deletes the export file too while un-exporting the volume.
Indeed, but the code is the following, only showing the two important functions:
function check_cmd_status()
{
if [ $1 != 0 ]
then
rm -rf $GANESHA_DIR/exports/export.$VOL.conf
exit 1
fi
}
function dynamic_export_remove()
{
removed_id=`cat $GANESHA_DIR/exports/export.$VOL.conf |\
grep Export_Id | cut -d -f8`
check_cmd_status `echo $?`
dbus-send --print-reply --system \
--dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr \
org.ganesha.nfsd.exportmgr.RemoveExport uint16:$removed_id
check_cmd_status `echo $?`
sed -i /$VOL.conf/d $CONF
rm -rf $GANESHA_DIR/exports/export.$VOL.conf
}
So, if the dbus-send fails the function check_cmd_status() just deletes
$GANESHA_DIR/exports/export.$VOL.conf and then quits the whole script, so the
last two lines (the sed and the rm) are never executed in
dynamic_export_remove()
# dbus-send --print-reply --system --dest=org.ganesha.nfsd
/org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2
Error org.freedesktop.DBus.Error.NoReply: Message did not receive a reply
(timeout by message bus)
So, there is a timeout and it fails completely.
Check if nfs-ganesha is still running. There was a bug in unexporting the
volume. Its been fixed recently in V2.3-dev, yet to be back-ported to
V2.2-stable branch.
https://review.gerrithub.io/#/c/236129/
Ah, I see! Yes, ganesha was still alive, otherwise the message would have been
different (something like ganesha is unknown by messagebus), and I have also
checked with service nfs-ganesha status. However, probably due to the bug
you’re mentioning, without issuing e.g. a DisplayExport I’m always getting
timeouts.
Thanks,
Alessandro
Thanks,
Soumya
In this case I think there is a bug in /usr/libexec/ganesha/dbus-send.sh,
since it blindly deletes the share config if the RemoveExport fails
(function check_cmd_status()), but leaves the %include inside ganesha.conf
as in the check_cmd_status() there is a runaway condition and the other
removal statements are then not executed. I believe the logic should be
fixed here, otherwise even a restart of the service will fail due to the bad
configuration.
What’s more worrying is the problem with the dbus. Issuing a DisplayExport
before the RemoveExport apparently fixes the problem, so something like this
always works:
# dbus-send --print-reply --system --dest=org.ganesha.nfsd
/org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.DisplayExport uint16:2
# dbus-send --print-reply --system --dest=org.ganesha.nfsd
/org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2
So, it’s like the DisplayExport is forcing someway a refresh that is needed
by the RemoveExport. Any idea why?
I’m using the latest version of ganesha 2.2.0, i.e. 2.2.0-3.
Thanks,
Alessandro
PS: sorry for reporting so many things in a few days :-)
--
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
smime.p7s
Description: S/MIME cryptographic signature
--
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Problems in /usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling exports from gluster
Alessandro De Salvo [alessandro.desa...@roma1.infn.it] wrote: What’s more worrying is the problem with the dbus. Issuing a DisplayExport before the RemoveExport apparently fixes the problem, so something like this always works: # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.DisplayExport uint16:2 # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2 So, it’s like the DisplayExport is forcing someway a refresh that is needed by the RemoveExport. Any idea why? I’m using the latest version of ganesha 2.2.0, i.e. 2.2.0-3. I used the same exact command above (the second one that removes an export) after restarting ganesha, and it just worked fine. I use GPFS FSAL (neither gluster nor VFS). Not sure why you need to use DisplayExport before using RemoveExport. Try to trace 'DBUS' component at FULL_DEBUG (or maybe everything at FULL_DEBUG) and post the log. The error you reported means we are NOT responding to the dbus message which is very odd! Reagrds, Malahal. -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Problems in /usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling exports from gluster
Hi Malahal, Il giorno 17/giu/2015, alle ore 19:51, Malahal Naineni mala...@us.ibm.com ha scritto: Alessandro De Salvo [alessandro.desa...@roma1.infn.it] wrote: What’s more worrying is the problem with the dbus. Issuing a DisplayExport before the RemoveExport apparently fixes the problem, so something like this always works: # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.DisplayExport uint16:2 # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2 So, it’s like the DisplayExport is forcing someway a refresh that is needed by the RemoveExport. Any idea why? I’m using the latest version of ganesha 2.2.0, i.e. 2.2.0-3. I used the same exact command above (the second one that removes an export) after restarting ganesha, and it just worked fine. I use GPFS FSAL (neither gluster nor VFS). Not sure why you need to use DisplayExport before using RemoveExport. Try to trace 'DBUS' component at FULL_DEBUG (or maybe everything at FULL_DEBUG) and post the log. The error you reported means we are NOT responding to the dbus message which is very odd! Indeed! This is why I was worried :-) I’m attaching the ganesha.log with NIV_FULL_DEBUG, you can see the restart at the end, due to my attempt of RemoveExport, which always gave me the same error as before. Thanks, Alessandro ganesha-20150617.log.gz Description: GNU Zip compressed data Reagrds, Malahal. smime.p7s Description: S/MIME cryptographic signature -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Problems in /usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling exports from gluster
I have no knowledge of dbus-send.sh, is it GLUSTER fsal specific? I don't see it in ganesha source code at all. I use ganesha_mgr command to show/delete/remove exports. Can you try it and see if that works? ganesha_mgr remove_export 2 should be the correct call in your case. Regards, Malahal. Alessandro De Salvo [alessandro.desa...@roma1.infn.it] wrote: Hi, when disabling exports from gluster 3.7.1, by using gluster vol set volume ganesha.enable off, I always get the following error: Error: Dynamic export addition/deletion failed. Please see log file for details This message is produced by the failure of /usr/libexec/ganesha/dbus-send.sh, and in fact if I manually perform the command to remove the share I see: # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2 Error org.freedesktop.DBus.Error.NoReply: Message did not receive a reply (timeout by message bus) So, there is a timeout and it fails completely. In this case I think there is a bug in /usr/libexec/ganesha/dbus-send.sh, since it blindly deletes the share config if the RemoveExport fails (function check_cmd_status()), but leaves the %include inside ganesha.conf as in the check_cmd_status() there is a runaway condition and the other removal statements are then not executed. I believe the logic should be fixed here, otherwise even a restart of the service will fail due to the bad configuration. What’s more worrying is the problem with the dbus. Issuing a DisplayExport before the RemoveExport apparently fixes the problem, so something like this always works: # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.DisplayExport uint16:2 # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2 So, it’s like the DisplayExport is forcing someway a refresh that is needed by the RemoveExport. Any idea why? I’m using the latest version of ganesha 2.2.0, i.e. 2.2.0-3. Thanks, Alessandro PS: sorry for reporting so many things in a few days :-) -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Problems in /usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling exports from gluster
On 06/17/2015 10:57 PM, Alessandro De Salvo wrote: Hi, when disabling exports from gluster 3.7.1, by using gluster vol set volume ganesha.enable off, I always get the following error: Error: Dynamic export addition/deletion failed. Please see log file for details This message is produced by the failure of /usr/libexec/ganesha/dbus-send.sh, and in fact if I manually perform the command to remove the share I see: you got it wrong. '/usr/libexec/ganesha/dbus-send.sh' is used by Gluster-CLI to unexport the volume gluster volume set volname ganesha.enable off which rightly deletes the export file too while un-exporting the volume. # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2 Error org.freedesktop.DBus.Error.NoReply: Message did not receive a reply (timeout by message bus) So, there is a timeout and it fails completely. Check if nfs-ganesha is still running. There was a bug in unexporting the volume. Its been fixed recently in V2.3-dev, yet to be back-ported to V2.2-stable branch. https://review.gerrithub.io/#/c/236129/ Thanks, Soumya In this case I think there is a bug in /usr/libexec/ganesha/dbus-send.sh, since it blindly deletes the share config if the RemoveExport fails (function check_cmd_status()), but leaves the %include inside ganesha.conf as in the check_cmd_status() there is a runaway condition and the other removal statements are then not executed. I believe the logic should be fixed here, otherwise even a restart of the service will fail due to the bad configuration. Yes. I see that the sed -i /$VOL.conf/d $CONF is placed after the check_cmd_status. I shall send a fix upstream in a related bug. But dynamic export removal will fail in three cases, 1. nfs-ganesha is not running. 2. The export file that is particular to that volume is somehow deleted before you perform the removal. It does depend on that file to get the export ID. 3. The bug that Soumya pointed out. If it is failing consistently, there could be something that you are missing. If you can send the exact sequence of sequence of steps that you have executed, I can help you with it. Ideally after exporting a particular volume, you'll see an entry in the /etc/ganesha/ganesha.conf file and the export file in /etc/ganesha/exports dir. If you have this in place and nfs-ganesha running, then dynamic export removal should work just fine. Meghana What’s more worrying is the problem with the dbus. Issuing a DisplayExport before the RemoveExport apparently fixes the problem, so something like this always works: # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.DisplayExport uint16:2 # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2 So, it’s like the DisplayExport is forcing someway a refresh that is needed by the RemoveExport. Any idea why? I’m using the latest version of ganesha 2.2.0, i.e. 2.2.0-3. Thanks, Alessandro PS: sorry for reporting so many things in a few days :-) -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
