On 06/18/2015 07:55 PM, Alessandro De Salvo wrote: > Hi Meghana, > >> Il giorno 18/giu/2015, alle ore 16:06, Meghana Madhusudhan >> <mmadh...@redhat.com> ha scritto: >> >> Hi Allesandro, >> >> I need the following output from you, >> >> 1. After you execute ganesha.enable on command. >> ps aux | grep ganesha > > root 6699 94.0 1.1 953488 193272 ? Ssl Jun17 1137:36 > /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N > NIV_EVENT -p /var/run/ganesha.nfsd.pid > root 22263 0.0 0.0 112644 960 pts/0 S+ 16:14 0:00 grep > --color=auto ganesha > > (both nodes of the cluster show the same, apart from the PIDs, of course) > >> showmount -e localhost > > Export list for node1: > /atlas-backup-01 (everyone) > > The same output is shown on the other node. > >> cat /etc/ganesha/ganesha.conf > > # Ganesha options file > %include "/etc/ganesha/ganesha-opts.conf" > %include "/etc/ganesha/exports/export.atlas-backup-01.conf" > > > The first include just point to a different file with the following lines, to > change the rquota port > > NFS_Core_Param { > #Use a non-privileged port for RQuota > Rquota_Port = 4501; > } > > >> cat "/etc/ganesha/exports/export.VOLNAME.conf" or >> "/usr/..../export.VOLNAME.conf” > > # WARNING : Using Gluster CLI will overwrite manual > # changes made to this file. To avoid it, edit the > # file, copy it over to all the NFS-Ganesha nodes > # and run ganesha-ha.sh --refresh-config. > EXPORT{ > Export_Id= 2 ; > Path = "/atlas-backup-01"; > FSAL { > name = GLUSTER; > hostname="localhost"; > volume="atlas-backup-01"; > } > Access_type = RW; > Disable_ACL = true; > Squash="No_root_squash"; > Pseudo="/atlas-backup-01"; > Protocols = "3", "4" ; > Transports = "UDP","TCP"; > SecType = "sys"; > } > > >> tail /var/log/ganesha.log or wherever the ganesha log is. > > 17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] > nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully > 17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] > nfs_Start_threads :THREAD :EVENT :admin thread was started successfully > 17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] > nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully > 17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[reaper] > nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE > 17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] > nfs_Start_threads :THREAD :EVENT :General fridge was started successfully > 17/06/2015 20:01:01 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] > nfs_start :NFS STARTUP :EVENT > :------------------------------------------------- > 17/06/2015 20:01:01 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] > nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED > 17/06/2015 20:01:01 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] > nfs_start :NFS STARTUP :EVENT > :------------------------------------------------- > 17/06/2015 20:02:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[reaper] > nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE > 18/06/2015 16:13:50 : epoch 5581b5da : node2 : > ganesha.nfsd-26983[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT > :Volume atlas-backup-01 exported at : '/' > > >> >> 2. gluster vol set VOLNAME ganesha.enable off >> >> Output of the same as above. > > # gluster vol set atlas-backup-01 ganesha.enable off > volume set: failed: Dynamic export addition/deletion failed. Please see log > file for details > > # ps aux | grep ganesha > root 32033 0.0 0.0 112640 960 pts/0 S+ 16:20 0:00 grep > --color=auto ganesha > > [This is due to the fact that the export file was deleted by the config still > references it, so ganesha fails to resume] > That would mean NFS-Ganesha process has crashed due the bug which I had mentioned earlier. The fix is merged into V2.3-dev branch and will get back-ported to V2.2-stable branch this week.
> # cat /etc/ganesha/ganesha.conf > # Ganesha options file > %include "/etc/ganesha/ganesha-opts.conf" > %include "/etc/ganesha/exports/export.atlas-backup-01.conf" > > # cat /etc/ganesha/exports/export.atlas-backup-01.conf > cat: /etc/ganesha/exports/export.atlas-backup-01.conf: No such file or > directory > > > # tail /var/log/ganesha.log > 18/06/2015 16:19:20 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] > glusterfs_create_export :FSAL :EVENT :Volume atlas-backup-01 exported at : '/' > 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] > lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed > for proper quota management in FSAL > 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] > lower_my_caps :NFS STARTUP :EVENT :currenty set capabilities are: = > cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap+ep > 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] > nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs > 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] > nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized > 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] > nfs4_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 60 > 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] > nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory > (/var/run/ganesha) already exists > 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] > nfs_rpc_cb_init_ccache :NFS STARTUP :WARN > :gssd_refresh_krb5_machine_credential failed (2:2) > 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] > nfs_Start_threads :THREAD :EVENT :Starting delayed executor. > 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] > nfs_Start_threads :THREAD :EVENT :9P/TCP dispatcher thread was started > successfully > 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[_9p_disp] > _9p_dispatcher_thread :9P DISP :EVENT :9P dispatcher started > 18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] > nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully > 18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] > nfs_Start_threads :THREAD :EVENT :admin thread was started successfully > 18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] > nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully > 18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[reaper] > nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE > 18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] > nfs_Start_threads :THREAD :EVENT :General fridge was started successfully > 18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] > nfs_start :NFS STARTUP :EVENT > :------------------------------------------------- > 18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] > nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED > 18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] > nfs_start :NFS STARTUP :EVENT > :------------------------------------------------- > 18/06/2015 16:20:22 : epoch 5582d3a6 : node2 : ganesha.nfsd-31542[main] main > :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version > /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 > 22:13:18 on node2 > 18/06/2015 16:20:23 : epoch 5582d3a7 : node2 : ganesha.nfsd-31550[main] main > :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version > /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 > 22:13:18 on node2 > 18/06/2015 16:20:23 : epoch 5582d3a7 : node2 : ganesha.nfsd-31557[main] main > :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version > /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 > 22:13:18 on node2 > 18/06/2015 16:20:23 : epoch 5582d3a7 : node2 : ganesha.nfsd-31562[main] main > :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version > /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 > 22:13:18 on node2 > 18/06/2015 16:20:23 : epoch 5582d3a7 : node2 : ganesha.nfsd-31567[main] main > :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version > /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 > 22:13:18 on node2 > 18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31908[main] main > :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version > /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 > 22:13:18 on node2 > 18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31932[main] main > :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version > /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 > 22:13:18 on node2 > 18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31949[main] main > :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version > /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 > 22:13:18 on node2 > 18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31956[main] main > :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version > /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 > 22:13:18 on node2 > 18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31971[main] main > :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version > /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 > 22:13:18 on node2 > > > > I have not enabled the FULL_DEBUG but if you need it I can do it. > >> >> Thanks for your perseverance :) > > I have to thank you for the help! :-) > Cheers, > > Alessandro > >> >> Meghana >> >> ----- Original Message ----- >> From: "Alessandro De Salvo" <alessandro.desa...@roma1.infn.it> >> To: "Meghana Madhusudhan" <mmadh...@redhat.com> >> Cc: gluster-us...@gluster.org, nfs-ganesha-devel@lists.sourceforge.net, >> "Soumya Koduri" <skod...@redhat.com> >> Sent: Thursday, June 18, 2015 7:24:55 PM >> Subject: Re: [Nfs-ganesha-devel] Problems in >> /usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling >> exports from gluster >> >> Hi Meghana, >> >>> Il giorno 18/giu/2015, alle ore 07:04, Meghana Madhusudhan >>> <mmadh...@redhat.com> ha scritto: >>> >>> >>> >>> >>> On 06/17/2015 10:57 PM, Alessandro De Salvo wrote: >>>> Hi, >>>> when disabling exports from gluster 3.7.1, by using gluster vol set >>>> <volume> ganesha.enable off, I always get the following error: >>>> >>>> Error: Dynamic export addition/deletion failed. Please see log file for >>>> details >>>> >>>> This message is produced by the failure of >>>> /usr/libexec/ganesha/dbus-send.sh, and in fact if I manually perform the >>>> command to remove the share I see: >>> you got it wrong. '/usr/libexec/ganesha/dbus-send.sh' is used by >>> Gluster-CLI to unexport the volume "gluster volume set <volname> >>> ganesha.enable off" which rightly deletes the export file too while >>> un-exporting the volume. >>> >>>> >>>> # dbus-send --print-reply --system --dest=org.ganesha.nfsd >>>> /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport >>>> uint16:2 >>>> Error org.freedesktop.DBus.Error.NoReply: Message did not receive a reply >>>> (timeout by message bus) >>>> >>>> So, there is a timeout and it fails completely. >>> Check if nfs-ganesha is still running. There was a bug in unexporting >>> the volume. Its been fixed recently in V2.3-dev, yet to be back-ported >>> to V2.2-stable branch. >>> https://review.gerrithub.io/#/c/236129/ >>> >>> Thanks, >>> Soumya >>> >>>> In this case I think there is a bug in /usr/libexec/ganesha/dbus-send.sh, >>>> since it blindly deletes the share config if the RemoveExport fails >>>> (function check_cmd_status()), but leaves the %include inside ganesha.conf >>>> as in the check_cmd_status() there is a runaway condition and the other >>>> removal statements are then not executed. I believe the logic should be >>>> fixed here, otherwise even a restart of the service will fail due to the >>>> bad configuration. >>> >>> Yes. I see that the "sed -i /$VOL.conf/d $CONF" is placed after the >>> check_cmd_status. I shall send a fix upstream in a related bug. But dynamic >>> export removal >>> will fail in three cases, >>> 1. nfs-ganesha is not running. >> >> no, it was running >> >>> 2. The export file that is particular to that volume is somehow deleted >>> before you perform the removal. It does depend on that file to get the >>> export ID. >> >> I tried to comment out the rm in check_cmd_status to avoid this race >> condition, but it did not solve the problems. >> >>> 3. The bug that Soumya pointed out. >> >> This might well be the real cause! >> >>> >>> If it is failing consistently, there could be something that you are >>> missing. If you can send the exact sequence of sequence of steps that you >>> have executed, >>> I can help you with it. >> >> Yes, it’s failing consistently, unless as I said I do a DisplayExport before >> the RemoveExport, in which case it always works. >> >>> >>> Ideally after exporting a particular volume, you'll see an entry in the >>> /etc/ganesha/ganesha.conf file and the export file in >>> "/etc/ganesha/exports" dir. >> >> And this works perfectly, I see them correctly. >> >>> If you have this in place and nfs-ganesha running, then dynamic export >>> removal should work just fine. >> >> But this is not, at least in my case. >> The command I’m using are just the following: >> >> gluster vol set <volume> ganesha.enable on >> gluster vol set <volume> ganesha.enable off >> >> It normally wait a few seconds between the two commands, to give time to >> ganesha to actually export the volume. >> The export is always failing as described, unless I add the DisplayExport in >> dbus-send.sh before RemoveExport. >> Many thanks for the help, >> >> Alessandro >> >> >>> >>> Meghana >>> >>> >>>> >>>> What’s more worrying is the problem with the dbus. Issuing a DisplayExport >>>> before the RemoveExport apparently fixes the problem, so something like >>>> this always works: >>>> >>>> # dbus-send --print-reply --system --dest=org.ganesha.nfsd >>>> /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.DisplayExport >>>> uint16:2 >>>> # dbus-send --print-reply --system --dest=org.ganesha.nfsd >>>> /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport >>>> uint16:2 >>>> >>>> So, it’s like the DisplayExport is forcing someway a refresh that is >>>> needed by the RemoveExport. Any idea why? >>>> I’m using the latest version of ganesha 2.2.0, i.e. 2.2.0-3. >>>> Thanks, >>>> >>>> Alessandro >>>> >>>> PS: sorry for reporting so many things in a few days :-) >>>> >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> >>>> >>>> >>>> _______________________________________________ >>>> Nfs-ganesha-devel mailing list >>>> Nfs-ganesha-devel@lists.sourceforge.net >>>> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel >>>> >>> >>> ------------------------------------------------------------------------------ >>> _______________________________________________ >>> Nfs-ganesha-devel mailing list >>> Nfs-ganesha-devel@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel >>> >> >> > ------------------------------------------------------------------------------ _______________________________________________ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel