Thank you Greg. It makes sense, that ceph-fuse will work this way. I am experiencing this problem for example when I run rsync on folder with lot of small files in ceph (~300GB in 10kB files). Ceph is during that very slow and everything crash like domino.
I will look into other ways of reading files from ceph. Most of the traffic is from webserver loading images - I could load these images with some script using some ceph library and implement simple timeout. -----Original Message----- From: Gregory Farnum [mailto:[email protected]] Sent: Wednesday, August 21, 2013 2:03 AM To: Petr Soukup Cc: [email protected] Subject: Re: [ceph-users] Ceph-fuse timeout? On Tue, Aug 20, 2013 at 4:56 PM, Petr Soukup <[email protected]> wrote: > I am using ceph filesystem through ceph-fuse to store product photos and most > of the time it works great. But if there is some problem on ceph server, my > connected clients start acting crazy. Load on all servers with mounted ceph > jumps very high, webserver and other services start to crash. > I think, that if ceph server is irresponsive, system creates some queue and > as a result everything starts to fail. Simple solution for this is this > command: > umount -fl /media/ceph && ceph-fuse /media/ceph After that everything > gets to normal in a few minutes. > > Is it possible to set some timeout to ceph-fuse or something? It is much > better if reading photo from ceph will cause error than if everything fails > at once. > > I am using ceph 0.61.7 (today upgraded to 0.61.8), 2x OSD, 1x MDS and 4x mon > on different servers with Centos 6.4. > I am going to try 0.67, but I think that my main problem is configuration of > ceph-fuse mount. I also tried newer kernel with support for mount ceph, but > the kernel itself wasn't very stable. What kind of problems are you seeing on the ceph server? In general what you're seeing is the result of ceph-fuse behaving optimistically: it expects that, if there is a problem on the cluster, then the problem will be dealt with shortly. So while it has lots of internal timeouts, it doesn't issue an error to clients unless it gets one back from a server. Doing otherwise in a data-safe fashion would be more or less impossible. If you want to add timeouts I bet you could find a library that will intercept system calls and let you put a timeout around them. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com _______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
