I did a bit more testing. 1. I tried on a newer kernel and was not able to recreate the problem, maybe it is that kernel bug you mentioned. Although its not an exact replica of the load. 2. I haven't tried the debug yet since I have to wait for the right moment.
One thing I realized and maybe it is not an issue is we are using a symlink to a folder in the ceph mount. ceph-fuse on /mnt/ceph type fuse.ceph-fuse (rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other) lrwxrwxrwx 1 root root metadata -> /mnt/ceph/DataCenter/metadata Not sure if that would create any issues. Anyway we are going to update the machine soon so, I can report if we keep having the issue. Thanks for your support, Scott On Mon, Mar 2, 2015 at 4:07 PM Scottix <[email protected]> wrote: > I'll try the following things and report back to you. > > 1. I can get a new kernel on another machine and mount to the CephFS and > see if I get the following errors. > 2. I'll run the debug and see if anything comes up. > > I'll report back to you when I can do these things. > > Thanks, > Scottie > > On Mon, Mar 2, 2015 at 4:04 PM Gregory Farnum <[email protected]> wrote: > >> I bet it's that permission issue combined with a minor bug in FUSE on >> that kernel, or maybe in the ceph-fuse code (but I've not seen it >> reported before, so I kind of doubt it). If you run ceph-fuse with >> "debug client = 20" it will output (a whole lot of) logging to the >> client's log file and you could see what requests are getting >> processed by the Ceph code and how it's responding. That might let you >> narrow things down. It's certainly not any kind of timeout. >> -Greg >> >> On Mon, Mar 2, 2015 at 3:57 PM, Scottix <[email protected]> wrote: >> > 3 Ceph servers on Ubuntu 12.04.5 - kernel 3.13.0-29-generic >> > >> > We have an old server that we compiled the ceph-fuse client on >> > Suse11.4 - kernel 2.6.37.6-0.11 >> > This is the only mount we have right now. >> > >> > We don't have any problems reading the files and the directory shows >> full >> > 775 permissions and doing a second ls fixes the problem. >> > >> > On Mon, Mar 2, 2015 at 3:51 PM Bill Sanders <[email protected]> >> wrote: >> >> >> >> Forgive me if this is unhelpful, but could it be something to do with >> >> permissions of the directory and not Ceph at all? >> >> >> >> http://superuser.com/a/528467 >> >> >> >> Bill >> >> >> >> On Mon, Mar 2, 2015 at 3:47 PM, Gregory Farnum <[email protected]> >> wrote: >> >>> >> >>> On Mon, Mar 2, 2015 at 3:39 PM, Scottix <[email protected]> wrote: >> >>> > We have a file system running CephFS and for a while we had this >> issue >> >>> > when >> >>> > doing an ls -la we get question marks in the response. >> >>> > >> >>> > -rw-r--r-- 1 wwwrun root 14761 Feb 9 16:06 >> >>> > data.2015-02-08_00-00-00.csv.bz2 >> >>> > -????????? ? ? ? ? ? >> >>> > data.2015-02-09_00-00-00.csv.bz2 >> >>> > >> >>> > If we do another directory listing it show up fine. >> >>> > >> >>> > -rw-r--r-- 1 wwwrun root 14761 Feb 9 16:06 >> >>> > data.2015-02-08_00-00-00.csv.bz2 >> >>> > -rw-r--r-- 1 wwwrun root 13675 Feb 10 15:21 >> >>> > data.2015-02-09_00-00-00.csv.bz2 >> >>> > >> >>> > It hasn't been a problem but just wanted to see if this is an issue, >> >>> > could >> >>> > the attributes be timing out? We do have a lot of files in the >> >>> > filesystem so >> >>> > that could be a possible bottleneck. >> >>> >> >>> Huh, that's not something I've seen before. Are the systems you're >> >>> doing this on the same? What distro and kernel version? Is it reliably >> >>> one of them showing the question marks, or does it jump between >> >>> systems? >> >>> -Greg >> >>> >> >>> > >> >>> > We are using the ceph-fuse mount. >> >>> > ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578) >> >>> > We are planning to do the update soon to 87.1 >> >>> > >> >>> > Thanks >> >>> > Scottie >> >>> > >> >>> > >> >>> > _______________________________________________ >> >>> > ceph-users mailing list >> >>> > [email protected] >> >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >>> > >> >>> _______________________________________________ >> >>> ceph-users mailing list >> >>> [email protected] >> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> >> >> >> > >> >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
