So a little further down the road I have destroyed/rebuilt the gluster volume and restarted glusterd on each of the nodes. Shortly after the instances directory is created and the console.log file is placed, I receive the following fuse errors on the client:
[client-rpc-fops.c:873:client3_3_writev_cbk] 0-novashared-client-4: remote operation failed: Invalid argument [2014-03-19 19:06:35.326487] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-novashared-client-5: remote operation failed: Invalid argument [2014-03-19 19:06:35.328750] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 106504: WRITE => -1 (Invalid argument) [2014-03-19 19:06:35.328855] W [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse: 106505: FTRUNCATE() ERR => -1 (Bad file descriptor) [2014-03-19 19:06:35.329506] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 106507: WRITE => -1 (Bad file descriptor) [2014-03-19 19:06:35.329775] W [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse: 106508: FTRUNCATE() ERR => -1 (Bad file descriptor) [2014-03-19 19:06:35.330495] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 106510: WRITE => -1 (Bad file descriptor) [2014-03-19 19:06:35.330758] W [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse: 106511: FTRUNCATE() ERR => -1 (Bad file descriptor) [2014-03-19 19:06:35.331538] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 106513: WRITE => -1 (Bad file descriptor) [2014-03-19 19:06:35.331792] W [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse: 106514: FTRUNCATE() ERR => -1 (Bad file descriptor) One one of the servers the normal messages apply: [2014-03-13 21:47:33.735042] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0 (version: 3.4.2) [2014-03-18 21:17:42.763044] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0 [2014-03-18 21:17:42.763076] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0 [2014-03-18 21:17:42.763101] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0 [2014-03-18 21:50:02.206635] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0 (version: 3.4.2) [2014-03-18 21:50:23.037886] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0 (version: 3.4.2) [2014-03-18 21:50:23.051419] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0 [2014-03-18 21:50:23.051446] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0 [2014-03-18 21:50:23.051466] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0 [2014-03-18 21:56:22.215866] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0 [2014-03-18 21:56:22.215898] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0 [2014-03-18 21:56:22.215923] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0 [2014-03-18 21:57:04.821397] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0 (version: 3.4.2) [2014-03-19 16:48:32.135593] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0 [2014-03-19 16:48:32.135629] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0 [2014-03-19 16:48:32.135652] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0 [2014-03-19 16:48:47.712973] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0 (version: 3.4.2) [2014-03-19 18:07:44.332240] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0 [2014-03-19 18:07:44.332272] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0 [2014-03-19 18:07:44.332296] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0 [2014-03-19 19:00:51.369205] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-21755-2014/03/19-19:00:51:286018-novashared-client-0-0 (version: 3.4.2) These are all on 3.4.2 with Ubuntu 13.10. -ryan On Mar 19, 2014, at 11:15 AM, Carlos Capriotti <[email protected]> wrote: > This rings a bell... > > I mapped a remote nfs share to one of my nodes and mapped the local nfs > share, to the "local" gluster volume as well and started to copy (with cp) > files from one mount to the other. > > Somewhere down the line all stalled, the server was irresponsive, even > restarting glusterd solved nothing. Had to reboot the nodes. > > CentOS 6.5 and Gluster 3.4.2. > > Since then, I haven't tried those copies, but, I also, I am now using the > native glusterfs client just in case. > > > > > On Wed, Mar 19, 2014 at 3:51 PM, Ray Powell <[email protected]> wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA512 >> >> I have not seen that with the fuse client. I had it happen repeatedly >> when we were using gluster/NFS for /var/lib/nova/instances and/or >> /var/lib/nova/instances/_base. The access to instances-XXXXXX/disk >> (or its qcow2 backing image) would hang completely or be very slow >> under heavy load. The VM interprets the delay as a problem on >> /dev/vda and remounts RO. >> >> >> On 3/18/14, 5:04 PM, Aydelott, Ryan M. wrote: >>> I was wondering if anyone had seen similar issues before, when >>> booting a vm with a root volume on local disk things work fine. >>> >>> When I try to do the same with /var/lib/nova/instances mounted via >>> gluster/fuse I get the following: http://paste.ubuntu.com/7116594/ >>> >>> I have of course insured perms were correct with chown -R nova:nova >>> /var/lib/nova/instances, additionally app armor is completely >>> unloaded/disabled. >>> >>> Has anyone encountered this before? Gluster version is 3.4.2 >>> >>> >>> -ryan >>> >>> >>> >>> >>> _______________________________________________ Gluster-users >>> mailing list [email protected] >>> http://supercolony.gluster.org/mailman/listinfo/gluster-users >>> >> >> - -- >> Raymond Powell >> Systems Administrator >> Laboratory for Advanced Computing >> University of Chicago, KCBD 10160 >> 900 East 57th Street >> Chicago, IL 60637 >> -----BEGIN PGP SIGNATURE----- >> Version: GnuPG/MacGPG2 v2.0.22 (Darwin) >> Comment: GPGTools - http://gpgtools.org >> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ >> >> iQIcBAEBCgAGBQJTKa8HAAoJEBIjAczqLCW8fYMQAIMsxPVG0n+ujRJeCYvphUy9 >> lHe7bkTLiJIkMs51hzXCfsqT3tEkvixYbjaAfjuHTdg0q5sK4zu+5MUM4KqG7oZy >> pYMzX2VrYnkTqQRFTOQbzntSvKlDjuSebuGpbI8sCxfgOR6/25tuzcJDCvn7sp4B >> YxVFaLZx5Xd6U24BHT9MVVJgAFXCVoalj2XMy+WHGhM07VvEXHxf62oW2VymLHOm >> eYiQGpjDWwb218/6NtJ2z47xr7HIQF1sW1Hhu/YRsnNzAYPh1FL3NwvnkFvNwGdg >> huTFpQegV9bmg78A3eEnIEEjDm1haX/E1NOo2bHdRYF2+Fwz+rHgkQJxFfEHxOV/ >> HDylUIPeMue9qEgttRg345NuypoVrBN3sFGf72UOfgwwElr+Z0t83y6uquTcNfOD >> lWIPa7d0vyLWfm6+mjR1hDHsLx6H/Y0MyAKFgYUIerdO8+TgAnLsv4FfBGlbm+Xa >> sutQxYPxBeAlSnKXmwrP9keB1ttiMV8IptHbuM5cQna5jdPeouwpluquZsP30bpA >> o/a7nPaogBx+X6BsM4aBjRLSjqtu+mywNh1AqdY1dIpvDtD9lXrO5L+Z2SLZUDeF >> JvFWkp8Epdn0k7vn6WKT6RZvjSji9WgTIdbFrKEjfEqAz/+9UbrG2D/vYL2TlgAe >> +UhAq6+74ie1MokNssB8 >> =un7j >> -----END PGP SIGNATURE----- >> _______________________________________________ >> Gluster-users mailing list >> [email protected] >> http://supercolony.gluster.org/mailman/listinfo/gluster-users >> > _______________________________________________ > Gluster-users mailing list > [email protected] > http://supercolony.gluster.org/mailman/listinfo/gluster-users
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ Gluster-users mailing list [email protected] http://supercolony.gluster.org/mailman/listinfo/gluster-users
