Re: [Gluster-users] NFS problem

Fri, 10 Jun 2011 00:23:43 -0700 

We'll need the crash stack trace also.


Christopher Anderlik wrote:

here are our logs when nfs is crashing....
[2011-06-10 08:54:14.900049] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: f8851fc2, ACCESS: NFS: 0(Call completed successfully.), POSIX: 0(Success) [2011-06-10 08:54:14.902002] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: f9851fc2, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:14.902037] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:14.902062] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: f9851fc2, GETATTR: args: FH: hashcount 3, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 74c48fb3-d065-462b-83a9-e4558b042465 [2011-06-10 08:54:14.920579] D [afr-transaction.c:976:afr_post_nonblocking_inodelk_cbk] 0-ksc-replicate-0: Non blocking inodelks done. Proceeding to FOP [2011-06-10 08:54:14.921099] D [client-lk.c:442:delete_granted_locks_fd] 0-ksc-client-0: Number of locks cleared=0 [2011-06-10 08:54:14.921155] D [client-lk.c:442:delete_granted_locks_fd] 0-ksc-client-1: Number of locks cleared=0 [2011-06-10 08:54:14.932629] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: f9851fc2, GETATTR: NFS: 0(Call completed successfully.), POSIX: 0(Success) [2011-06-10 08:54:14.932863] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: fa851fc2, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4 [2011-06-10 08:54:14.932890] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS [2011-06-10 08:54:14.932907] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: fa851fc2, ACCESS: args: FH: hashcount 3, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 74c48fb3-d065-462b-83a9-e4558b042465 [2011-06-10 08:54:14.961700] D [socket.c:193:__socket_rwv] 0-ksc-client-0: EOF from peer 10.0.1.198:24031 [2011-06-10 08:54:14.961741] W [socket.c:1494:__socket_proto_state_machine] 0-ksc-client-0: reading from socket failed. Error (Transport endpoint is not connected), peer (10.0.1.198:24031) [2011-06-10 08:54:14.961757] D [socket.c:1768:socket_event_handler] 0-transport: disconnecting now [2011-06-10 08:54:14.961858] E [rpc-clnt.c:338:saved_frames_unwind] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(rpc_clnt_notify+0x158) [0x7f140e8c0acc] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x101) [0x7f140e8c006a] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(saved_frames_destroy+0x1c) [0x7f140e8bfb78]))) 0-ksc-client-0: forced unwinding frame type(GlusterFS 3.1) op(SETATTR(38)) called at 2011-06-10 08:54:14.920644 [2011-06-10 08:54:14.961880] I [client3_1-fops.c:1640:client3_1_setattr_cbk] 0-ksc-client-0: remote operation failed: Transport endpoint is not connected [2011-06-10 08:54:14.961880] I [client3_1-fops.c:1640:client3_1_setattr_cbk] 0-ksc-client-0: remote operation failed: Transport endpoint is not connected [2011-06-10 08:54:14.961940] I [client.c:1883:client_rpc_notify] 0-ksc-client-0: disconnected [2011-06-10 08:54:14.988468] W [socket.c:204:__socket_rwv] 0-ksc-client-1: readv failed (Connection reset by peer) [2011-06-10 08:54:14.988490] W [socket.c:1494:__socket_proto_state_machine] 0-ksc-client-1: reading from socket failed. Error (Connection reset by peer), peer (10.0.1.199:24027) [2011-06-10 08:54:14.988501] D [socket.c:1768:socket_event_handler] 0-transport: disconnecting now [2011-06-10 08:54:14.988501] D [socket.c:1768:socket_event_handler] 0-transport: disconnecting now [2011-06-10 08:54:14.988551] E [rpc-clnt.c:338:saved_frames_unwind] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(rpc_clnt_notify+0x158) [0x7f140e8c0acc] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x101) [0x7f140e8c006a] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(saved_frames_destroy+0x1c) [0x7f140e8bfb78]))) 0-ksc-client-1: forced unwinding frame type(GlusterFS 3.1) op(SETATTR(38)) called at 2011-06-10 08:54:14.920655 [2011-06-10 08:54:14.988568] I [client3_1-fops.c:1640:client3_1_setattr_cbk] 0-ksc-client-1: remote operation failed: Transport endpoint is not connected [2011-06-10 08:54:14.988568] I [client3_1-fops.c:1640:client3_1_setattr_cbk] 0-ksc-client-1: remote operation failed: Transport endpoint is not connected [2011-06-10 08:54:14.988599] D [client.c:77:client_submit_request] 0-ksc-client-0: connection in disconnected state [2011-06-10 08:54:14.988630] W [client3_1-fops.c:4379:client3_1_xattrop] 0-ksc-client-0: failed to send the fop: Transport endpoint is not connected [2011-06-10 08:54:14.988657] D [name.c:157:client_fill_address_family] 0-ksc-client-1: address-family not specified, guessing it to be inet/inet6 [2011-06-10 08:54:14.991719] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.0.1.199 (port-24007) for hostname: 10.0.1.199 and port: 24007 [2011-06-10 08:54:14.991786] I [socket.c:2272:socket_submit_request] 0-ksc-client-1: not connected (priv->connected = 0) [2011-06-10 08:54:14.991803] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-ksc-client-1: failed to submit rpc-request (XID: 0x70357x Program: GlusterFS 3.1, ProgVers: 310, Proc: 33) to rpc-transport (ksc-client-1) [2011-06-10 08:54:14.991819] D [afr-lk-common.c:409:transaction_lk_op] 0-ksc-replicate-0: lk op is for a transaction [2011-06-10 08:54:14.991833] D [client.c:77:client_submit_request] 0-ksc-client-0: connection in disconnected state [2011-06-10 08:54:14.991845] W [client3_1-fops.c:4735:client3_1_inodelk] 0-ksc-client-0: failed to send the fop: Transport endpoint is not connected [2011-06-10 08:54:14.991862] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-ksc-client-1: failed to submit rpc-request (XID: 0x70358x Program: GlusterFS 3.1, ProgVers: 310, Proc: 29) to rpc-transport (ksc-client-1) [2011-06-10 08:54:14.991876] I [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-ksc-client-1: remote operation failed: Transport endpoint is not connected [2011-06-10 08:54:14.991904] D [nfs3-helpers.c:2477:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: c71a69ff, CREATE: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: hashcount 4, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid f3e21f99-0aed-4042-8b81-16a804497b5b [2011-06-10 08:54:14.991945] D [client.c:129:client_submit_request] 0-ksc-client-1: rpc_clnt_submit failed [2011-06-10 08:54:14.991967] D [client.c:129:client_submit_request] 0-ksc-client-1: rpc_clnt_submit failed [2011-06-10 08:54:14.992014] E [rpc-clnt.c:338:saved_frames_unwind] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(rpc_clnt_notify+0x158) [0x7f140e8c0acc] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x101) [0x7f140e8c006a] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(saved_frames_destroy+0x1c) [0x7f140e8bfb78]))) 0-ksc-client-1: forced unwinding frame type(GlusterFS 3.1) op(RELEASE(41)) called at 2011-06-10 08:54:14.921180 [2011-06-10 08:54:14.992046] E [rpc-clnt.c:338:saved_frames_unwind] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(rpc_clnt_notify+0x158) [0x7f140e8c0acc] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x101) [0x7f140e8c006a] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(saved_frames_destroy+0x1c) [0x7f140e8bfb78]))) 0-ksc-client-1: forced unwinding frame type(GlusterFS 3.1) op(STAT(1)) called at 2011-06-10 08:54:14.932952 [2011-06-10 08:54:14.992060] I [client3_1-fops.c:411:client3_1_stat_cbk] 0-ksc-client-1: remote operation failed: Transport endpoint is not connected [2011-06-10 08:54:14.992073] D [client.c:77:client_submit_request] 0-ksc-client-0: connection in disconnected state [2011-06-10 08:54:14.992091] W [client3_1-fops.c:2658:client3_1_stat] 0-ksc-client-0: failed to send the fop Transport endpoint is not connected [2011-06-10 08:54:14.992101] D [afr-inode-read.c:204:afr_stat_cbk] 0-ksc-replicate-0: /pimp/htdocs/mountpoints.sh: all subvolumes tried, going out [2011-06-10 08:54:14.992115] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: fa851fc2, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:14.992181] I [client.c:1883:client_rpc_notify] 0-ksc-client-1: disconnected [2011-06-10 08:54:14.992197] E [afr-common.c:2546:afr_notify] 0-ksc-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-06-10 08:54:15.5285] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: fb851fc2, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:15.5312] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.5312] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.5344] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: fb851fc2, GETATTR: args: FH: hashcount 3, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 74c48fb3-d065-462b-83a9-e4558b042465 [2011-06-10 08:54:15.5381] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /pimp/htdocs/mountpoints.sh: no child is up [2011-06-10 08:54:15.5399] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: fb851fc2, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:15.5783] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: fc851fc2, Ver: 2, Program: 100003, ProgVers: 3, Proc: 3 [2011-06-10 08:54:15.5810] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - LOOKUP [2011-06-10 08:54:15.5833] D [nfs3-helpers.c:2304:nfs3_log_fh_entry_call] 0-nfs-nfsv3: XID: fc851fc2, LOOKUP: args: FH: hashcount 2, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid ff18fc56-c352-4caf-b98d-4eff71494acc, name: mountpoints.sh [2011-06-10 08:54:15.5897] D [nfs3.c:1080:nfs3_fresh_lookup] 0-nfs-nfsv3: inode needs fresh lookup [2011-06-10 08:54:15.5954] D [nfs3-helpers.c:2477:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: fc851fc2, LOOKUP: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: hashcount 0, exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000 [2011-06-10 08:54:15.11858] E [socket.c:1685:socket_connect_finish] 0-ksc-client-1: connection to 10.0.1.199:24027 failed (Connection refused) [2011-06-10 08:54:15.11886] D [socket.c:289:__socket_disconnect] 0-ksc-client-1: shutdown() returned -1. Transport endpoint is not connected [2011-06-10 08:54:15.11921] D [socket.c:193:__socket_rwv] 0-ksc-client-1: EOF from peer 10.0.1.199:24027 [2011-06-10 08:54:15.11938] D [socket.c:1494:__socket_proto_state_machine] 0-ksc-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.0.1.199:24027) [2011-06-10 08:54:15.11954] D [socket.c:1768:socket_event_handler] 0-transport: disconnecting now [2011-06-10 08:54:15.158995] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: e1f47a24, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:15.159038] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.159038] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.159065] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: e1f47a24, GETATTR: args: FH: hashcount 6, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 9b62d8e0-cf19-487e-b56a-9ac980294801 [2011-06-10 08:54:15.159105] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /vertrieb/htdocs/vertrieb_imrich/fetischtopliste.de/www/button.php: no child is up [2011-06-10 08:54:15.159126] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: e1f47a24, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:15.159501] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: e2f47a24, Ver: 2, Program: 100003, ProgVers: 3, Proc: 3 [2011-06-10 08:54:15.159528] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - LOOKUP [2011-06-10 08:54:15.159550] D [nfs3-helpers.c:2304:nfs3_log_fh_entry_call] 0-nfs-nfsv3: XID: e2f47a24, LOOKUP: args: FH: hashcount 5, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid dd60aa24-78fa-4509-abee-acf32f12542f, name: button.php [2011-06-10 08:54:15.159604] D [nfs3.c:1080:nfs3_fresh_lookup] 0-nfs-nfsv3: inode needs fresh lookup [2011-06-10 08:54:15.159654] D [nfs3-helpers.c:2477:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: e2f47a24, LOOKUP: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: hashcount 0, exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000 [2011-06-10 08:54:15.160014] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: e3f47a24, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:15.160041] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.160080] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: e3f47a24, GETATTR: args: FH: hashcount 5, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid dd60aa24-78fa-4509-abee-acf32f12542f [2011-06-10 08:54:15.160113] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /vertrieb/htdocs/vertrieb_imrich/fetischtopliste.de/www: no child is up [2011-06-10 08:54:15.160130] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: e3f47a24, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:15.160532] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: e4f47a24, Ver: 2, Program: 100003, ProgVers: 3, Proc: 3 [2011-06-10 08:54:15.160554] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - LOOKUP [2011-06-10 08:54:15.160554] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - LOOKUP [2011-06-10 08:54:15.160569] D [nfs3-helpers.c:2304:nfs3_log_fh_entry_call] 0-nfs-nfsv3: XID: e4f47a24, LOOKUP: args: FH: hashcount 4, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 434fc86c-5bed-4a38-b919-8822d38bd1a3, name: www [2011-06-10 08:54:15.160611] D [nfs3.c:1080:nfs3_fresh_lookup] 0-nfs-nfsv3: inode needs fresh lookup [2011-06-10 08:54:15.160657] D [nfs3-helpers.c:2477:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: e4f47a24, LOOKUP: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: hashcount 0, exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000 [2011-06-10 08:54:15.192962] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: c81a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:15.193000] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.193019] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: c81a69ff, GETATTR: args: FH: hashcount 3, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 61273392-07cb-4427-bb48-2cf869031802 [2011-06-10 08:54:15.193050] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /tomorrowwinners.com/typo3temp/locks: no child is up [2011-06-10 08:54:15.193069] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: c81a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:15.193526] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: c91a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 3 [2011-06-10 08:54:15.193547] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - LOOKUP [2011-06-10 08:54:15.193564] D [nfs3-helpers.c:2304:nfs3_log_fh_entry_call] 0-nfs-nfsv3: XID: c91a69ff, LOOKUP: args: FH: hashcount 2, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 96012827-9101-40a1-b540-b6c47fe49e59, name: locks [2011-06-10 08:54:15.193607] D [nfs3.c:1080:nfs3_fresh_lookup] 0-nfs-nfsv3: inode needs fresh lookup [2011-06-10 08:54:15.193654] D [nfs3-helpers.c:2477:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: c91a69ff, LOOKUP: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: hashcount 0, exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000 [2011-06-10 08:54:15.193987] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: ca1a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:15.194015] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.194037] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: ca1a69ff, GETATTR: args: FH: hashcount 2, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 96012827-9101-40a1-b540-b6c47fe49e59 [2011-06-10 08:54:15.194070] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /tomorrowwinners.com/typo3temp: no child is up [2011-06-10 08:54:15.194086] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: ca1a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:15.194566] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: cb1a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 3 [2011-06-10 08:54:15.194587] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - LOOKUP [2011-06-10 08:54:15.194614] D [nfs3-helpers.c:2304:nfs3_log_fh_entry_call] 0-nfs-nfsv3: XID: cb1a69ff, LOOKUP: args: FH: hashcount 1, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 7fecc27e-a8de-4fe5-adce-c5e00d35c7ac, name: typo3temp [2011-06-10 08:54:15.194659] D [nfs3.c:1080:nfs3_fresh_lookup] 0-nfs-nfsv3: inode needs fresh lookup [2011-06-10 08:54:15.194703] D [nfs3-helpers.c:2477:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: cb1a69ff, LOOKUP: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: hashcount 0, exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000 [2011-06-10 08:54:15.396934] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: cc1a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:15.396989] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.396989] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.397017] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: cc1a69ff, GETATTR: args: FH: hashcount 1, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 7fecc27e-a8de-4fe5-adce-c5e00d35c7ac [2011-06-10 08:54:15.397064] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /tomorrowwinners.com: no child is up [2011-06-10 08:54:15.397087] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: cc1a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:15.397441] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: cd1a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 3 [2011-06-10 08:54:15.397468] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - LOOKUP [2011-06-10 08:54:15.397492] D [nfs3-helpers.c:2304:nfs3_log_fh_entry_call] 0-nfs-nfsv3: XID: cd1a69ff, LOOKUP: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001, name: tomorrowwinners.com [2011-06-10 08:54:15.397551] D [nfs3.c:1080:nfs3_fresh_lookup] 0-nfs-nfsv3: inode needs fresh lookup [2011-06-10 08:54:15.397600] D [nfs3-helpers.c:2477:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: cd1a69ff, LOOKUP: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: hashcount 0, exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000 [2011-06-10 08:54:15.397953] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: ce1a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4 [2011-06-10 08:54:15.397981] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS [2011-06-10 08:54:15.398004] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: ce1a69ff, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:15.398038] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /: no child is up [2011-06-10 08:54:15.398055] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: ce1a69ff, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:15.600726] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: cf1a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:15.600769] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.600769] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.600789] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: cf1a69ff, GETATTR: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:15.600857] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: cf1a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:15.601525] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d01a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4 [2011-06-10 08:54:15.601553] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS [2011-06-10 08:54:15.601576] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d01a69ff, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:15.601627] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /: no child is up [2011-06-10 08:54:15.601646] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d01a69ff, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:15.725647] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: fd851fc2, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4 [2011-06-10 08:54:15.725689] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS [2011-06-10 08:54:15.725689] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS [2011-06-10 08:54:15.725719] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: fd851fc2, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:15.725749] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /: no child is up [2011-06-10 08:54:15.725769] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: fd851fc2, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:15.726209] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: fe851fc2, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4 [2011-06-10 08:54:15.726236] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS [2011-06-10 08:54:15.726252] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: fe851fc2, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:15.726276] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /: no child is up [2011-06-10 08:54:15.726292] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: fe851fc2, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:15.804546] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d11a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:15.804580] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.804580] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:15.804606] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d11a69ff, GETATTR: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:15.804658] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d11a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:15.805060] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d21a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4 [2011-06-10 08:54:15.805088] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS [2011-06-10 08:54:15.805111] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d21a69ff, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:15.805143] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /: no child is up [2011-06-10 08:54:15.805159] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d21a69ff, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:16.8639] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d31a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:16.8698] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:16.20711] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d31a69ff, GETATTR: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:16.20821] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d31a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:16.21517] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d41a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4 [2011-06-10 08:54:16.21556] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS [2011-06-10 08:54:16.21575] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d41a69ff, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:16.21606] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /: no child is up [2011-06-10 08:54:16.21625] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d41a69ff, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:16.225038] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d51a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:16.225096] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:16.225096] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:16.225118] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d51a69ff, GETATTR: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:16.225193] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d51a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:16.225595] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d61a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4 [2011-06-10 08:54:16.225622] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS [2011-06-10 08:54:16.225639] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d61a69ff, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:16.225669] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /: no child is up [2011-06-10 08:54:16.225688] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d61a69ff, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:16.243210] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d71a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4 [2011-06-10 08:54:16.243232] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS [2011-06-10 08:54:16.243232] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS [2011-06-10 08:54:16.243249] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d71a69ff, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:16.243274] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /: no child is up [2011-06-10 08:54:16.243291] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d71a69ff, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:16.243670] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d81a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:16.243697] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:16.243717] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d81a69ff, GETATTR: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:16.243756] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d81a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:16.428628] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d91a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1 [2011-06-10 08:54:16.428689] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:16.428689] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR [2011-06-10 08:54:16.428711] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d91a69ff, GETATTR: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 [2011-06-10 08:54:16.428787] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d91a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected) [2011-06-10 08:54:16.429129] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: da1a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4 [2011-06-10 08:54:16.429157] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS [2011-06-10 08:54:16.429182] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: da1a69ff, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001 












Am 09.06.2011 17:09, schrieb anthony garnier:

Hi,

I got the same problem as Juergen,
My volume is a simple replicated volume with 2 host and GlusterFS 3.2.0

Volume Name: poolsave
Type: Replicate
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: ylal2950:/soft/gluster-data
Brick2: ylal2960:/soft/gluster-data
Options Reconfigured:
diagnostics.brick-log-level: DEBUG
network.ping-timeout: 20
performance.cache-size: 512MB
nfs.port: 2049

I'm running this command :

I get those error :
tar: ./uvs00: owner not changed
tar: could not stat ./uvs00/log/0906uvsGESEC.log
tar: ./uvs00: group not changed
tar: could not stat ./uvs00/log/0306uvsGESEC.log
tar: ./uvs00/log: Input/output error
cannot change back?: Unknown error 526
tar: ./uvs00/log: owner not changed
tar: ./uvs00/log: group not changed
tar: tape blocksize error

And then I tried to "ls" in gluster mount :
/bin/ls: .: Input/output error

only way is to restart the volume


Here is the logfile in Debug mod :


Given volfile:
+------------------------------------------------------------------------------+ 
1: volume poolsave-client-0
2: type protocol/client
3: option remote-host ylal2950
4: option remote-subvolume /soft/gluster-data
5: option transport-type tcp
6: option ping-timeout 20
7: end-volume
8:
9: volume poolsave-client-1
10: type protocol/client
11: option remote-host ylal2960
12: option remote-subvolume /soft/gluster-data
13: option transport-type tcp
14: option ping-timeout 20
15: end-volume
16:
17: volume poolsave-replicate-0
18: type cluster/replicate
19: subvolumes poolsave-client-0 poolsave-client-1
20: end-volume
21:
22: volume poolsave-write-behind
23: type performance/write-behind
24: subvolumes poolsave-replicate-0
25: end-volume
26:
27: volume poolsave-read-ahead
28: type performance/read-ahead
29: subvolumes poolsave-write-behind
30: end-volume
31:
32: volume poolsave-io-cache
33: type performance/io-cache
34: option cache-size 512MB
35: subvolumes poolsave-read-ahead
36: end-volume
37:
38: volume poolsave-quick-read
39: type performance/quick-read
40: option cache-size 512MB
41: subvolumes poolsave-io-cache
42: end-volume
43:
44: volume poolsave-stat-prefetch
45: type performance/stat-prefetch
46: subvolumes poolsave-quick-read
47: end-volume
48:
49: volume poolsave
50: type debug/io-stats
51: option latency-measurement off
52: option count-fop-hits off
53: subvolumes poolsave-stat-prefetch
54: end-volume
55:
56: volume nfs-server
57: type nfs/server
58: option nfs.dynamic-volumes on
59: option rpc-auth.addr.poolsave.allow *
60: option nfs3.poolsave.volume-id 71e0dabf-4620-4b6d-b138-3266096b93b6
61: option nfs.port 2049
62: subvolumes poolsave
63: end-volume
+------------------------------------------------------------------------------+ [2011-06-09 16:52:23.709018] I [rpc-clnt.c:1531:rpc_clnt_reconfig] 0-poolsave-client-0: changing 
port to 24014 (from 0)
[2011-06-09 16:52:23.709211] I [rpc-clnt.c:1531:rpc_clnt_reconfig] 0-poolsave-client-1: changing 
port to 24011 (from 0)
[2011-06-09 16:52:27.716417] I [client-handshake.c:1080:select_server_supported_programs] 0-poolsave-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310) [2011-06-09 16:52:27.716650] I [client-handshake.c:913:client_setvolume_cbk] 0-poolsave-client-0: Connected to 10.68.217.85:24014, attached to remote volume '/soft/gluster-data'. [2011-06-09 16:52:27.716679] I [afr-common.c:2514:afr_notify] 0-poolsave-replicate-0: Subvolume 
'poolsave-client-0' came back up; going online.
[2011-06-09 16:52:27.717020] I [afr-common.c:836:afr_fresh_lookup_cbk] 0-poolsave-replicate-0: added 
root inode
[2011-06-09 16:52:27.729719] I [client-handshake.c:1080:select_server_supported_programs] 0-poolsave-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310) [2011-06-09 16:52:27.730014] I [client-handshake.c:913:client_setvolume_cbk] 0-poolsave-client-1: Connected to 10.68.217.86:24011, attached to remote volume '/soft/gluster-data'. [2011-06-09 17:01:35.537084] W [stat-prefetch.c:178:sp_check_and_create_inode_ctx] (-->/usr/local/lib/glusterfs/3.2.0/xlator/nfs/server.so(nfs_fop_mkdir+0x1cc) [0x2aaaab3b88fc] (-->/usr/local/lib/glusterfs/3.2.0/xlator/debug/io-stats.so(io_stats_mkdir+0x151) [0x2aaaab2948e1] (-->/usr/local/lib/glusterfs/3.2.0/xlator/performance/stat-prefetch.so(sp_mkdir+0xd2) [0x2aaaab1856c2]))) 0-poolsave-stat-prefetch: stat-prefetch context is present in inode (ino:0 gfid:00000000-0000-0000-0000-000000000000) when it is supposed to be not present [2011-06-09 17:01:35.546601] W [stat-prefetch.c:178:sp_check_and_create_inode_ctx] (-->/usr/local/lib/glusterfs/3.2.0/xlator/nfs/server.so(nfs_fop_create+0x1db) [0x2aaaab3b95bb] (-->/usr/local/lib/glusterfs/3.2.0/xlator/debug/io-stats.so(io_stats_create+0x165) [0x2aaaab294ad5] (-->/usr/local/lib/glusterfs/3.2.0/xlator/performance/stat-prefetch.so(sp_create+0xbc) [0x2aaaab185c9c]))) 0-poolsave-stat-prefetch: stat-prefetch context is present in inode (ino:0 gfid:00000000-0000-0000-0000-000000000000) when it is supposed to be not present [2011-06-09 17:01:35.569755] I [client3_1-fops.c:547:client3_1_rmdir_cbk] 0-poolsave-client-0: 
remote operation failed: Directory not empty
[2011-06-09 17:01:35.569881] I [client3_1-fops.c:547:client3_1_rmdir_cbk] 0-poolsave-client-1: 
remote operation failed: Directory not empty
[2011-06-09 17:01:35.579674] W [stat-prefetch.c:178:sp_check_and_create_inode_ctx] (-->/usr/local/lib/glusterfs/3.2.0/xlator/nfs/server.so(nfs_fop_mkdir+0x1cc) [0x2aaaab3b88fc] (-->/usr/local/lib/glusterfs/3.2.0/xlator/debug/io-stats.so(io_stats_mkdir+0x151) [0x2aaaab2948e1] (-->/usr/local/lib/glusterfs/3.2.0/xlator/performance/stat-prefetch.so(sp_mkdir+0xd2) [0x2aaaab1856c2]))) 0-poolsave-stat-prefetch: stat-prefetch context is present in inode (ino:0 gfid:00000000-0000-0000-0000-000000000000) when it is supposed to be not present [2011-06-09 17:01:35.587907] W [stat-prefetch.c:178:sp_check_and_create_inode_ctx] (-->/usr/local/lib/glusterfs/3.2.0/xlator/nfs/server.so(nfs_fop_create+0x1db) [0x2aaaab3b95bb] (-->/usr/local/lib/glusterfs/3.2.0/xlator/debug/io-stats.so(io_stats_create+0x165) [0x2aaaab294ad5] (-->/usr/local/lib/glusterfs/3.2.0/xlator/performance/stat-prefetch.so(sp_create+0xbc) [0x2aaaab185c9c]))) 0-poolsave-stat-prefetch: stat-prefetch context is present in inode (ino:0 gfid:00000000-0000-0000-0000-000000000000) when it is supposed to be not present [2011-06-09 17:01:35.612918] W [stat-prefetch.c:178:sp_check_and_create_inode_ctx] (-->/usr/local/lib/glusterfs/3.2.0/xlator/nfs/server.so(nfs_fop_create+0x1db) [0x2aaaab3b95bb] (-->/usr/local/lib/glusterfs/3.2.0/xlator/debug/io-stats.so(io_stats_create+0x165) [0x2aaaab294ad5] (-->/usr/local/lib/glusterfs/3.2.0/xlator/performance/stat-prefetch.so(sp_create+0xbc) [0x2aaaab185c9c]))) 0-poolsave-stat-prefetch: stat-prefetch context is present in inode (ino:0 gfid:00000000-0000-0000-0000-000000000000) when it is supposed to be not present [2011-06-09 17:01:35.645357] W [stat-prefetch.c:178:sp_check_and_create_inode_ctx] (-->/usr/local/lib/glusterfs/3.2.0/xlator/nfs/server.so(nfs_fop_create+0x1db) [0x2aaaab3b95bb] (-->/usr/local/lib/glusterfs/3.2.0/xlator/debug/io-stats.so(io_stats_create+0x165) [0x2aaaab294ad5] (-->/usr/local/lib/glusterfs/3.2.0/xlator/performance/stat-prefetch.so(sp_create+0xbc) [0x2aaaab185c9c]))) 0-poolsave-stat-prefetch: stat-prefetch context is present in inode (ino:0 gfid:00000000-0000-0000-0000-000000000000) when it is supposed to be not present [2011-06-09 17:01:35.660873] I [client3_1-fops.c:547:client3_1_rmdir_cbk] 0-poolsave-client-0: 
remote operation failed: Directory not empty
[2011-06-09 17:01:35.660955] I [client3_1-fops.c:547:client3_1_rmdir_cbk] 0-poolsave-client-1: 
remote operation failed: Directory not empty
[2011-06-09 17:01:35.665933] I [client3_1-fops.c:547:client3_1_rmdir_cbk] 0-poolsave-client-0: 
remote operation failed: Directory not empty
[2011-06-09 17:01:35.666057] I [client3_1-fops.c:547:client3_1_rmdir_cbk] 0-poolsave-client-1: 
remote operation failed: Directory not empty
[2011-06-09 17:01:35.671199] I [client3_1-fops.c:547:client3_1_rmdir_cbk] 0-poolsave-client-0: 
remote operation failed: Directory not empty
[2011-06-09 17:01:35.671241] I [client3_1-fops.c:547:client3_1_rmdir_cbk] 0-poolsave-client-1: 
remote operation failed: Directory not empty
[2011-06-09 17:01:35.680959] W [stat-prefetch.c:178:sp_check_and_create_inode_ctx] (-->/usr/local/lib/glusterfs/3.2.0/xlator/nfs/server.so(nfs_fop_create+0x1db) [0x2aaaab3b95bb] (-->/usr/local/lib/glusterfs/3.2.0/xlator/debug/io-stats.so(io_stats_create+0x165) [0x2aaaab294ad5] (-->/usr/local/lib/glusterfs/3.2.0/xlator/performance/stat-prefetch.so(sp_create+0xbc) [0x2aaaab185c9c]))) 0-poolsave-stat-prefetch: stat-prefetch context is present in inode (ino:0 gfid:00000000-0000-0000-0000-000000000000) when it is supposed to be not present [2011-06-09 17:01:35.715633] W [stat-prefetch.c:178:sp_check_and_create_inode_ctx] (-->/usr/local/lib/glusterfs/3.2.0/xlator/nfs/server.so(nfs_fop_create+0x1db) [0x2aaaab3b95bb] (-->/usr/local/lib/glusterfs/3.2.0/xlator/debug/io-stats.so(io_stats_create+0x165) [0x2aaaab294ad5] (-->/usr/local/lib/glusterfs/3.2.0/xlator/performance/stat-prefetch.so(sp_create+0xbc) [0x2aaaab185c9c]))) 0-poolsave-stat-prefetch: stat-prefetch context is present in inode (ino:0 gfid:00000000-0000-0000-0000-000000000000) when it is supposed to be not present [2011-06-09 17:01:35.732798] I [client3_1-fops.c:547:client3_1_rmdir_cbk] 0-poolsave-client-0: 
remote operation failed: Permission denied
[2011-06-09 17:01:35.733044] I [client3_1-fops.c:547:client3_1_rmdir_cbk] 0-poolsave-client-1: 
remote operation failed: Permission denied
[2011-06-09 17:01:35.750009] W [stat-prefetch.c:178:sp_check_and_create_inode_ctx] (-->/usr/local/lib/glusterfs/3.2.0/xlator/nfs/server.so(nfs_fop_create+0x1db) [0x2aaaab3b95bb] (-->/usr/local/lib/glusterfs/3.2.0/xlator/debug/io-stats.so(io_stats_create+0x165) [0x2aaaab294ad5] (-->/usr/local/lib/glusterfs/3.2.0/xlator/performance/stat-prefetch.so(sp_create+0xbc) [0x2aaaab185c9c]))) 0-poolsave-stat-prefetch: stat-prefetch context is present in inode (ino:0 gfid:00000000-0000-0000-0000-000000000000) when it is supposed to be not present [2011-06-09 17:01:35.784610] W [socket.c:1494:__socket_proto_state_machine] 0-poolsave-client-0: reading from socket failed. Error (Transport endpoint is not connected), peer (10.68.217.85:24014) 
[2011-06-09 17:01:35.784745] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2ab58145f7f9]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2ab58145ef8e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2ab58145eefe]))) 0-poolsave-client-0: forced unwinding frame type(GlusterFS 3.1) op(SETATTR(38)) called at 2011-06-09 17:01:35.752080 [2011-06-09 17:01:35.784770] I [client3_1-fops.c:1640:client3_1_setattr_cbk] 0-poolsave-client-0: 
remote operation failed: Transport endpoint is not connected
[2011-06-09 17:01:35.784811] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2ab58145f7f9]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2ab58145ef8e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2ab58145eefe]))) 0-poolsave-client-0: forced unwinding frame type(GlusterFS 3.1) op(STAT(1)) called at 2011-06-09 17:01:35.752414 [2011-06-09 17:01:35.784828] I [client3_1-fops.c:411:client3_1_stat_cbk] 0-poolsave-client-0: remote 
operation failed: Transport endpoint is not connected
[2011-06-09 17:01:35.784875] I [client.c:1883:client_rpc_notify] 0-poolsave-client-0: disconnected [2011-06-09 17:01:35.785400] W [socket.c:204:__socket_rwv] 0-poolsave-client-1: readv failed 
(Connection reset by peer)
[2011-06-09 17:01:35.785435] W [socket.c:1494:__socket_proto_state_machine] 0-poolsave-client-1: reading from socket failed. Error (Connection reset by peer), peer (10.68.217.86:24011) 
[2011-06-09 17:01:35.785496] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2ab58145f7f9]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2ab58145ef8e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2ab58145eefe]))) 0-poolsave-client-1: forced unwinding frame type(GlusterFS 3.1) op(SETATTR(38)) called at 2011-06-09 17:01:35.752089 [2011-06-09 17:01:35.785516] I [client3_1-fops.c:1640:client3_1_setattr_cbk] 0-poolsave-client-1: 
remote operation failed: Transport endpoint is not connected
[2011-06-09 17:01:35.785542] W [client3_1-fops.c:4379:client3_1_xattrop] 0-poolsave-client-0: failed 
to send the fop: Transport endpoint is not connected
[2011-06-09 17:01:35.817662] I [socket.c:2272:socket_submit_request] 0-poolsave-client-1: not 
connected (priv->connected = 0)
[2011-06-09 17:01:35.817698] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-poolsave-client-1: failed to submit rpc-request (XID: 0x576x Program: GlusterFS 3.1, ProgVers: 310, Proc: 33) to rpc-transport 
(poolsave-client-1)
[2011-06-09 17:01:35.817721] W [client3_1-fops.c:4735:client3_1_inodelk] 0-poolsave-client-0: failed 
to send the fop: Transport endpoint is not connected
[2011-06-09 17:01:35.817744] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-poolsave-client-1: failed to submit rpc-request (XID: 0x577x Program: GlusterFS 3.1, ProgVers: 310, Proc: 29) to rpc-transport 
(poolsave-client-1)
[2011-06-09 17:01:35.817780] I [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-poolsave-client-1: 
remote operation failed: Transport endpoint is not connected
[2011-06-09 17:01:35.817897] E [rpc-clnt.c:338:saved_frames_unwind]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2ab58145f7f9]
(-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2ab58145ef8e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2ab58145eefe]))) 0-poolsave-client-1: forced unwinding frame type(GlusterFS 3.1) op(STAT(1)) called at 2011-06-09 17:01:35.784870 [2011-06-09 17:01:35.817918] I [client3_1-fops.c:411:client3_1_stat_cbk] 0-poolsave-client-1: remote 
operation failed: Transport endpoint is not connected
[2011-06-09 17:01:35.817969] I [client.c:1883:client_rpc_notify] 0-poolsave-client-1: disconnected [2011-06-09 17:01:35.817988] E [afr-common.c:2546:afr_notify] 0-poolsave-replicate-0: All subvolumes 
are down. Going offline until atleast one of them comes back up.
[2011-06-09 17:01:35.818007] E [socket.c:1685:socket_connect_finish] 0-poolsave-client-1: connection 
to 10.68.217.86:24011 failed (Connection refused)
[2011-06-09 17:01:35.818606] I [afr.h:838:AFR_LOCAL_INIT] 0-poolsave-replicate-0: no subvolumes up [2011-06-09 17:01:35.819129] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /uvs00/log: 
no child is up
[2011-06-09 17:01:35.819354] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /uvs00/log: 
no child is up
[2011-06-09 17:01:35.820090] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /uvs00: no 
child is up
[2011-06-09 17:01:35.820760] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /: no child is up [2011-06-09 17:01:35.821212] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /: no child is up [2011-06-09 17:01:35.821600] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /: no child is up [2011-06-09 17:01:35.822123] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /: no child is up [2011-06-09 17:01:35.822511] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /: no child is up [2011-06-09 17:01:35.822975] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /: no child is up [2011-06-09 17:01:35.823286] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /: no child is up [2011-06-09 17:01:35.823583] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /: no child is up [2011-06-09 17:01:35.823857] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /: no child is up [2011-06-09 17:01:47.518006] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /: no child is up [2011-06-09 17:01:49.39204] E [socket.c:1685:socket_connect_finish] 0-poolsave-client-0: connection 
to 10.68.217.85:24014 failed (Connection refused)
[2011-06-09 17:01:49.136932] I [afr-inode-read.c:270:afr_stat] 0-poolsave-replicate-0: /: no child is up 



 > Message: 7
 > Date: Thu, 9 Jun 2011 12:56:39 +0530
 > From: Shehjar Tikoo <shehj...@gluster.com>
 > Subject: Re: [Gluster-users] Glusterfs 3.2.0 NFS Problem
 > To: J?rgen Winkler <juergen.wink...@xidras.com>
 > Cc: gluster-users@gluster.org
 > Message-ID: <4df075af.3040...@gluster.com>
 > Content-Type: text/plain; charset="us-ascii"; format=flowed
 >
> This can happen if all your servers were unreachable for a few seconds. The > situation must have rectified during the restart. We could confirm if you 
 > change the log level on nfs to DEBUG and send us the log.
 >
 > Thanks
 > -Shehjar
 >
 > Ju"rgen Winkler wrote:
 > > Hi,
 > >
 > > i noticed a strange behavior with NFS and Glusterfs 3.2.0 , 3 of our
 > > Servers are loosing the Mount but when you restart the Volume on the
 > > Server it works again without a remount.
 > >
> > On the server i noticed this entries in the Glusterfs/Nfs log-file when 
 > > the mount on the Client becomes unavailable :
 > >
 > > [2011-06-08 14:37:02.568693] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:02.569212] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:02.611910] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:02.624477] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:04.288272] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:04.296150] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:04.309247] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:04.320939] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:04.321786] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:04.333609] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:04.334089] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:04.344662] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:04.352666] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:04.354195] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:04.360446] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:04.369331] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:04.471556] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:04.480013] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:05.639700] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:05.652535] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:07.578469] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:07.588949] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:07.590395] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:07.591414] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:07.591932] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:07.592596] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:07.639317] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:07.652919] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:09.332435] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:09.340622] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:09.349360] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:09.349550] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:09.360445] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:09.369497] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:09.369752] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:09.382097] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > > [2011-06-08 14:37:09.382387] I [afr-inode-read.c:270:afr_stat]
 > > 0-ksc-replicate-0: /: no child is up
 > >
 > >
 > > Thx for the help
 > >
 > > _______________________________________________
 > > Gluster-users mailing list
 > > Gluster-users@gluster.org
 > > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
 >
 >
 >
 > ------------------------------
 >
 > _______________________________________________
 > Gluster-users mailing list
 > Gluster-users@gluster.org
 > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
 >
 >
 > End of Gluster-users Digest, Vol 38, Issue 14
 > *********************************************



_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users




_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Reply via email to