Re: [ceph-users] CephFS Attributes Question Marks

On Tue, Sep 29, 2015 at 9:51 PM, Scottix wrote: > I'm positive the client I sent you the log is 94. We do have one client > still on 87. > which version of kernel are you using? I found a kernel bug which can cause this issue in 4.1 and later kernels. Regards Yan, Zheng >

Re: [ceph-users] CephFS Attributes Question Marks

OpenSuse 12.1 3.1.10-1.29-desktop On Wed, Sep 30, 2015, 5:34 AM Yan, Zheng wrote: > On Tue, Sep 29, 2015 at 9:51 PM, Scottix wrote: > >> I'm positive the client I sent you the log is 94. We do have one client >> still on 87. >> > which version of kernel

Re: [ceph-users] CephFS Attributes Question Marks

Hmm, so apparently a similar bug was fixed in 0.87: Scott, can you confirm that your *clients* were 0.94 (not just the servers)? Thanks, John On Tue, Sep 29, 2015 at 11:56 AM, John Spray wrote: > Ah, this is a nice clear log! > > I've described the bug here: >

Re: [ceph-users] CephFS Attributes Question Marks

Ah, this is a nice clear log! I've described the bug here: http://tracker.ceph.com/issues/13271 In the short term, you may be able to mitigate this by increasing client_cache_size (on the client) if your RAM allows it. John On Tue, Sep 29, 2015 at 12:58 AM, Scottix wrote:

Re: [ceph-users] CephFS Attributes Question Marks

I did a bit more testing. 1. I tried on a newer kernel and was not able to recreate the problem, maybe it is that kernel bug you mentioned. Although its not an exact replica of the load. 2. I haven't tried the debug yet since I have to wait for the right moment. One thing I realized and maybe it

Re: [ceph-users] CephFS Attributes Question Marks

Ya we are not at 0.87.1 yet, possibly tomorrow. I'll let you know if it still reports the same. Thanks John, --Scottie On Tue, Mar 3, 2015 at 2:57 PM John Spray john.sp...@redhat.com wrote: On 03/03/2015 22:35, Scottix wrote: I was testing a little bit more and decided to run the

Re: [ceph-users] CephFS Attributes Question Marks

I was testing a little bit more and decided to run the cephfs-journal-tool I ran across some errors $ cephfs-journal-tool journal inspect 2015-03-03 14:18:54.453981 7f8e29f86780 -1 Bad entry start ptr (0x2aebf6) at 0x2aeb32279b 2015-03-03 14:18:54.539060 7f8e29f86780 -1 Bad entry start ptr

Re: [ceph-users] CephFS Attributes Question Marks

On 03/03/2015 22:35, Scottix wrote: I was testing a little bit more and decided to run the cephfs-journal-tool I ran across some errors $ cephfs-journal-tool journal inspect 2015-03-03 14:18:54.453981 7f8e29f86780 -1 Bad entry start ptr (0x2aebf6) at 0x2aeb32279b 2015-03-03

Re: [ceph-users] CephFS Attributes Question Marks

On 03/03/2015 22:57, John Spray wrote: On 03/03/2015 22:35, Scottix wrote: I was testing a little bit more and decided to run the cephfs-journal-tool I ran across some errors $ cephfs-journal-tool journal inspect 2015-03-03 14:18:54.453981 7f8e29f86780 -1 Bad entry start ptr (0x2aebf6)

[ceph-users] CephFS Attributes Question Marks

We have a file system running CephFS and for a while we had this issue when doing an ls -la we get question marks in the response. -rw-r--r-- 1 wwwrun root14761 Feb 9 16:06 data.2015-02-08_00-00-00.csv.bz2 -? ? ? ? ?? data.2015-02-09_00-00-00.csv.bz2 If we

Re: [ceph-users] CephFS Attributes Question Marks

Forgive me if this is unhelpful, but could it be something to do with permissions of the directory and not Ceph at all? http://superuser.com/a/528467 Bill On Mon, Mar 2, 2015 at 3:47 PM, Gregory Farnum g...@gregs42.com wrote: On Mon, Mar 2, 2015 at 3:39 PM, Scottix scot...@gmail.com wrote:

Re: [ceph-users] CephFS Attributes Question Marks

I bet it's that permission issue combined with a minor bug in FUSE on that kernel, or maybe in the ceph-fuse code (but I've not seen it reported before, so I kind of doubt it). If you run ceph-fuse with debug client = 20 it will output (a whole lot of) logging to the client's log file and you

Re: [ceph-users] CephFS Attributes Question Marks

On Mon, Mar 2, 2015 at 3:39 PM, Scottix scot...@gmail.com wrote: We have a file system running CephFS and for a while we had this issue when doing an ls -la we get question marks in the response. -rw-r--r-- 1 wwwrun root14761 Feb 9 16:06 data.2015-02-08_00-00-00.csv.bz2 -? ? ?

Re: [ceph-users] CephFS Attributes Question Marks

3 Ceph servers on Ubuntu 12.04.5 - kernel 3.13.0-29-generic We have an old server that we compiled the ceph-fuse client on Suse11.4 - kernel 2.6.37.6-0.11 This is the only mount we have right now. We don't have any problems reading the files and the directory shows full 775 permissions and doing

Re: [ceph-users] CephFS Attributes Question Marks

I'll try the following things and report back to you. 1. I can get a new kernel on another machine and mount to the CephFS and see if I get the following errors. 2. I'll run the debug and see if anything comes up. I'll report back to you when I can do these things. Thanks, Scottie On Mon, Mar