On Oct 5, 2007, at 10:49 AM, Sam Lang wrote:
The obvious and easy fix is to have bmi-tcp return true from
DROP_ADDR_QUERY for all address references. As far as I can tell,
the only thing we save by keeping them around is a little memory
allocation (the socket gets closed either way).
This suggested fix isn't right. The DEC_ADDR_REF which decrements
the refcount to zero, is invoked after sending the final response,
but that's usually before the client (in the case of the admin tools)
closes the connection. It looks like its the tcp_forget_addr in the
bmi method that needs to call back out to the bmi wrapper layer to
remove the reference from the list. I can call BMI_set_info(addr,
BMI_TCP_CLOSE_SOCKET) from tcp_forget_addr, but that seems a bit
backwards...
-sam
In the changes I've been working on to get multiple address support
in BMI, I've already replaced the linked list with a hashtable,
which wouldn't have made the problem go away, but the degradation
wouldn't have been quite as bad (may have made it harder to find,
actually). Maybe its time to add some profiling info (perf stats?)
to our basic list, queue and hash structures that would tell us how
big they're getting.
Anyway, thanks to all for contributing to the debugging process for
this one.
-sam
On Sep 26, 2007, at 6:00 PM, Sam Lang wrote:
Hi All,
I've been trying to debug a problem with PVFS, where performance
degrades slowly with a long-lived (weeks and months) PVFS volume.
The degradation is significant -- simple metadata operations are
an order of magnitude slower after a month or so. The behavior
turns out to only occur with the VFS and pvfs2-client daemon:
performance of the admin tools (pvfs2-touch, pvfs2-rm, etc.) to
the same set of servers remains good. Restarting the client
daemon also fixes the problem, suggesting that the long-lived open
sockets are somehow the cause. The slowness also appears to be at
the servers not the clients: the same kernel module and client
daemon to a different filesystem and set of servers doesn't
exhibit the performance degradation.
Also, I should mention that the system config is a little
different than usual. We have IO nodes mounting and unmounting
the PVFS volume (and stopping the client daemon) with each user's
job, which is fairly frequent, while on the login nodes, the
volume remains mounted for a long time (and where the performance
degrades).
Our hunch here is that epoll or our use of epoll on the servers is
somehow to blame. Maybe the file descriptors opened on the server
for pvfs2-client-core are getting pushed down further and further
into the epoll set, which for some reason is growing with new
connections coming and going. This might be the case if we were
failing to remove sockets from the set on disconnect, for
example. It doesn't look like that's happening though, at least
for normal disconnects.
Its a PITA to debug, because the servers have to remain running
for a long time (and the clients have to remain mounted) for the
problem to be visible. Rob suggested I use strace on the servers
to see what epoll was doing, and that showed some interesting
results. Basically, it looks like epoll_wait takes significantly
longer when clients are doing operations over the VFS, rather than
with the pvfs2 admin tools. Also, strace reported epoll_ctl(...,
EPOLL_CTL_ADD, ...)) getting called a few times, even for the VFS
ops, and in those cases its returning EEXISTS.
I noticed that we add a socket to the epoll set whenever we get a
new connection, or a read or write is posted (enqueue_operation),
but we only remove the socket from the epoll set on errors or
disconnects. So why are we adding it for reads and writes? Any
connected socket should already be in the set, no? I think this
may be why I'm seeing EEXISTS with strace.
Also, is it safe to check the error from epoll_ctl in
BMI_socket_collection_[add|remove]?
And finally, assuming PVFS is actually using epoll calls properly,
does anyone know of epoll bugs on a SUSE 2.6.5 kernel that would
cause epoll_ctl(..., EPOLL_CTL_DEL, ....) to not do what its meant
to? Googling epoll and SUSE 2.6.5 isn't turning up anything...
Thanks,
-sam
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
