Frank, Could you share your volume configuration (`gluster volume info <VOLNAME>`)?
-Krutika On Tue, Jul 12, 2016 at 3:44 PM, David Gossage <[email protected]> wrote: > > > On Tue, Jul 12, 2016 at 5:08 AM, Frank Rothenstein < > [email protected]> wrote: > >> Hi David, >> >> As I'm on gluster I use >> >> glusterfs.x86_64 3.7.12-2.el7 @centos-gluster37 >> >> I already tried to revert back to 3.7.11 but somehow this didn't help. >> >> Do you have more info on aio/caching? >> > > Not at the moment I really haven't had time to dig into it this week yet. > I'll quote Lindsay Mathieson 's response and maybe she can chime in if she > sees this chain. > > > -------------------------------------------------------------- > Hi David, I did a quick test this morning with Proxmox and 3.7.13 and was > able to get it working with the fuse mount *and* libgfapi. > > > One caveat - you *have* to enable qemu caching, either write-back or > write-through. 3.7.12 & 13 seem to now disable aio support, and qemu > requires that when caching is turned off. > > > There are setting for aio in gluster that I haven't played with yet. > ----------------------------------------------------------------- > > Did you revert client and server's to 3.7.11? It could be also that we > had 2 completely different issues. The only similarity being the recent > upgrades and Operation not permitted messages. > > >> FrankAm Dienstag, den 12.07.2016, 04:25 -0500 schrieb David Gossage: >> > >> > On Tue, Jul 12, 2016 at 4:06 AM, Frank Rothenstein <f.rothenstein@bod >> > den-kliniken.de> wrote: >> > > Hi all, >> > > >> > > I am stuck with a strange problem using ovirt/gluster on >> > > ZFSonLinux. >> > > >> > > I detached a data-volume form my Cluster an reimported it to change >> > > from POSIX-type/glusterfs to GlusterFS-Storage. When I try to >> > > attach it >> > > to the Cluster I always get a Sanlock error. I checked the logs and >> > > what I found were some "operation not permitted" >> > > errors in the brick logs: >> > > >> > > [2016-07-08 09:19:24.542046] E [MSGID: 113040] [posix- >> > > helpers.c:1684:__posix_fd_ctx_get] 0-datadom-posix: Failed to get >> > > anonymous real_path: >> > > /tank2/datadom/brick1/br/.glusterfs/c1/43/c1433bab-cd72-4395-ac62- >> > > b278670c333a _fd = -1 [No such file or directory] >> > > [2016-07-08 09:19:24.542103] W [MSGID: 113006] >> > > [posix.c:3010:posix_readv] 0-datadom-posix: pfd is NULL from >> > > fd=0x7f15f0003b48 [Operation not permitted] >> > > [2016-07-08 09:19:24.542142] E [MSGID: 115068] [server-rpc- >> > > fops.c:1486:server_readv_cbk] 0-datadom-server: 4836: READV -2 >> > > (c1433bab-cd72-4395-ac62-b278670c333a) ==> (Operation not >> > > permitted) >> > > [Operation not permitted] >> > > >> > > Anyone ever expierienced such problem? >> > > >> > > In fact I am able to create/change/delete files in the brick path >> > > itself and the mount points (NFS/Fuse). >> > > Ovirt itself seems to test the volumes with /usr/bin/dd, but uses >> > > iflag=direct. Using this parameter I'm also unable to create any >> > > file - >> > > is there any gluster-option that changed recently? Or is it ZFS, >> > > that >> > > introduced something that blocks now? >> > >> > What version of Gluster are you on? >> > >> > My errors aren't quite the same as yours, but I have a similar setup >> > >> > Ovirt 3.6.6/Gluster3.7.11 on ZFSonLinux >> > >> > Updating from 3.7.11 -> 12 or 13 this last weekend I had these issues >> > pop up for me which while not exactly the same did result >> > in [Operation not permitted] >> > >> > [2016-07-09 15:27:46.935694] I [fuse-bridge.c:4083:fuse_init] 0- >> > glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 >> > kernel 7.22 >> > [2016-07-09 15:27:49.555466] W [MSGID: 114031] [client-rpc- >> > fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-1: remote >> > operation failed [Operation not permitted] >> > [2016-07-09 15:27:49.556574] W [MSGID: 114031] [client-rpc- >> > fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-0: remote >> > operation failed [Operation not permitted] >> > [2016-07-09 15:27:49.556659] W [fuse-bridge.c:2227:fuse_readv_cbk] 0- >> > glusterfs-fuse: 80: READ => -1 gfid=deb61291-5176-4b81-8315- >> > 3f1cf8e3534d fd=0x7f5224002f68 (Operation not permitted) >> > [2016-07-09 15:27:59.612477] W [MSGID: 114031] [client-rpc- >> > fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-1: remote >> > operation failed [Operation not permitted] >> > [2016-07-09 15:27:59.613700] W [MSGID: 114031] [client-rpc- >> > fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-0: remote >> > operation failed [Operation not permitted] >> > [2016-07-09 15:27:59.613781] W [fuse-bridge.c:2227:fuse_readv_cbk] 0- >> > glusterfs-fuse: 168: READ => -1 gfid=deb61291-5176-4b81-8315- >> > 3f1cf8e3534d fd=0x7f5224002f68 (Operation not permitted) >> > >> > Another gluster user suggested that in 3.7.12/13 aio support was >> > disabled and I may need to change disk caching method. So far I have >> > just stayed at 3.7.11 until I have time to get a better test setup. >> > >> > > Thanks in advance, Frank >> > > >> > > >> > > >> > > >> > > >> > > ___________________________________________________________________ >> > > ___________ >> > > BODDEN-KLINIKEN Ribnitz-Damgarten GmbH >> > > Sandhufe 2 >> > > 18311 Ribnitz-Damgarten >> > > >> > > Telefon: 03821-700-0 >> > > Fax: 03821-700-240 >> > > >> > > E-Mail: [email protected] Internet: http://www.bodden-klini >> > > ken.de >> > > >> > > Sitz: Ribnitz-Damgarten, Amtsgericht: Stralsund, HRB 2919, Steuer- >> > > Nr.: 079/133/40188 >> > > Aufsichtsratsvorsitzende: Carmen Schröter, Geschäftsführer: Dr. >> > > Falko Milski >> > > >> > > Der Inhalt dieser E-Mail ist ausschließlich für den bezeichneten >> > > Adressaten bestimmt. Wenn Sie nicht der vorge- >> > > sehene Adressat dieser E-Mail oder dessen Vertreter sein sollten, >> > > beachten Sie bitte, dass jede Form der Veröf- >> > > fentlichung, Vervielfältigung oder Weitergabe des Inhalts dieser E- >> > > Mail unzulässig ist. Wir bitten Sie, sofort den >> > > Absender zu informieren und die E-Mail zu löschen. >> > > >> > > >> > > Bodden-Kliniken Ribnitz-Damgarten GmbH 2016 >> > > *** Virenfrei durch Kerio Mail Server und Sophos Antivirus *** >> > > >> > > _______________________________________________ >> > > Gluster-users mailing list >> > > [email protected] >> > > http://www.gluster.org/mailman/listinfo/gluster-users >> > >> >> >> >> >> >> >> ______________________________________________________________________________ >> BODDEN-KLINIKEN Ribnitz-Damgarten GmbH >> Sandhufe 2 >> 18311 Ribnitz-Damgarten >> >> Telefon: 03821-700-0 >> Fax: 03821-700-240 >> >> E-Mail: [email protected] Internet: http://www.bodden-kliniken.de >> >> Sitz: Ribnitz-Damgarten, Amtsgericht: Stralsund, HRB 2919, Steuer-Nr.: >> 079/133/40188 >> Aufsichtsratsvorsitzende: Carmen Schröter, Geschäftsführer: Dr. Falko >> Milski >> >> Der Inhalt dieser E-Mail ist ausschließlich für den bezeichneten >> Adressaten bestimmt. Wenn Sie nicht der vorge- >> sehene Adressat dieser E-Mail oder dessen Vertreter sein sollten, >> beachten Sie bitte, dass jede Form der Veröf- >> fentlichung, Vervielfältigung oder Weitergabe des Inhalts dieser E-Mail >> unzulässig ist. Wir bitten Sie, sofort den >> Absender zu informieren und die E-Mail zu löschen. >> >> >> Bodden-Kliniken Ribnitz-Damgarten GmbH 2016 >> *** Virenfrei durch Kerio Mail Server und Sophos Antivirus *** >> >> > > _______________________________________________ > Gluster-users mailing list > [email protected] > http://www.gluster.org/mailman/listinfo/gluster-users >
_______________________________________________ Gluster-users mailing list [email protected] http://www.gluster.org/mailman/listinfo/gluster-users
