Thanks all for commenting.
It turned out to be an easier problem.
This is an interactive login node with a lot of users logging in.
So, I put in lots of limits to prevent abuse of cpu time, number of
processes, open file limits, etc.
Account provisioning process mistakenly puts root into a group that has
limitations.
Unfortunately, the limits enforced by /etc/security/limits.conf is not
logged anywhere (so, the debugging process got a lot longer).
I found out the limit when I decided to probe into /proc/<PID>/limits
To ensure the proper limits, I put the following in mount.glusterfs:
ulimit -S -s 10240
ulimit -n 65536
ulimit -t unlimited
Thanks all for commenting.
Robin
On 12/17/12 10:59 AM, Robin, Robin wrote:
Yup. I rsync over to a Gluster mounted (and not the actual brick) via
client over the network.
Robin
From: "Washer, Bryan" <[email protected] <mailto:[email protected]>>
Date: Mon, 17 Dec 2012 10:47:38 -0500
To: Joe Julian <[email protected] <mailto:[email protected]>>,
"[email protected] <mailto:[email protected]>"
<[email protected] <mailto:[email protected]>>
Subject: Re: [Gluster-users] Transport endpoint
Just to make sure we don't miss the obviousŠwhen you say sync'd over to
the mount pointŠyou mean where you did a glusterfs mount and not eh actual
location of the brick on one of the mirrors in your replica.
Once you setup the volume and start it.you should NEVER write or delete
directly on the backend brick unless you really know what you are doing.
Bryan
-----Original Message-----
From: Joe Julian <[email protected] <mailto:[email protected]>>
Date: Monday, December 17, 2012 9:29 AM
To: "[email protected] <mailto:[email protected]>"
<[email protected] <mailto:[email protected]>>
Subject: Re: [Gluster-users] Transport endpoint
On 12/17/2012 06:56 AM, Robin, Robin wrote:
Hi,
I've got Gluster error: Transport endpoint not connected.
It came up twice after trying to rsync 2 TB filesystem over; it
reached about 1.8 TB and got the error.
Logs on the server side (on reverse time order):
[2012-12-15 00:53:24.747934] I
[server-helpers.c:629:server_connection_destroy]
0-RedhawkShared-server: destroyed connection of
mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-
client-0-0
[2012-12-15 00:53:24.743459] I [server-helpers.c:474:do_fd_cleanup]
0-RedhawkShared-server: fd cleanup on
/mkennedy/tramelot_nwfs/rpr3/rpr3/rpr3_sparky/matrix/.4d_ccnoesy.ucsf.QTQ
swL
[2012-12-15 00:53:24.743430] I
[server-helpers.c:330:do_lock_table_cleanup] 0-RedhawkShared-server:
finodelk released on
/mkennedy/tramelot_nwfs/rpr3/rpr3/rpr3_sparky/matrix/.4d_ccnoesy.ucsf.QTQ
swL
[2012-12-15 00:53:24.743400] I
[server-helpers.c:741:server_connection_put] 0-RedhawkShared-server:
Shutting down connection
mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-
client-0-0
[2012-12-15 00:53:24.743368] I [server.c:685:server_rpc_notify]
0-RedhawkShared-server: disconnecting connectionfrom
mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-
client-0-0
[2012-12-15 00:53:24.740055] W [socket.c:195:__socket_rwv]
0-tcp.RedhawkShared-server: readv failed (Connection reset by peer)
I can't find relevant logs on the client side.
From the logs, can we judge for sure that this is a network reset
problem ?
When you say, "I can't find relevant logs on the client side," do you
mean that you can't find the log, or that there's nothing in there from
around the same timestamp? The client log will be in /var/log/glusterfs
and will be named based on the mountpoint.
_______________________________________________
Gluster-users mailing list
[email protected] <mailto:[email protected]>
http://supercolony.gluster.org/mailman/listinfo/gluster-users
NOTICE: This email and any attachments may contain confidential and
proprietary information of NetSuite Inc. and is for the sole use of the
intended recipient for the stated purpose. Any improper use or
distribution is prohibited. If you are not the intended recipient,
please notify the sender; do not review, copy or distribute; and
promptly delete or destroy all transmitted information. Please note that
all communications and information transmitted through this email system
may be monitored by NetSuite or its agents and that all incoming email
is automatically scanned by a third party spam and filtering service.
_______________________________________________
Gluster-users mailing list
[email protected]
http://supercolony.gluster.org/mailman/listinfo/gluster-users
