Re: [ceph-users] CephFS Attributes Question Marks

2015-09-30 Thread Yan, Zheng 
On Tue, Sep 29, 2015 at 9:51 PM, Scottix  wrote:

> I'm positive the client I sent you the log is 94. We do have one client
> still on 87.
>
which version of kernel are you using? I found a kernel bug which can cause
this issue in 4.1 and later kernels.

Regards
Yan, Zheng



>
> On Tue, Sep 29, 2015, 6:42 AM John Spray  wrote:
>
>>
>> Hmm, so apparently a similar bug was fixed in 0.87: Scott, can you
>> confirm that your *clients* were 0.94 (not just the servers)?
>>
>> Thanks,
>> John
>>
>> On Tue, Sep 29, 2015 at 11:56 AM, John Spray  wrote:
>>
>>> Ah, this is a nice clear log!
>>>
>>> I've described the bug here:
>>> http://tracker.ceph.com/issues/13271
>>>
>>> In the short term, you may be able to mitigate this by increasing
>>> client_cache_size (on the client) if your RAM allows it.
>>>
>>> John
>>>
>>> On Tue, Sep 29, 2015 at 12:58 AM, Scottix  wrote:
>>>
 I know this is an old one but I got a log in ceph-fuse for it.
 I got this on OpenSuse 12.1
 3.1.10-1.29-desktop

 Using ceph-fuse
 ceph version 0.94.3 (95cefea9fd9ab740263bf8bb4796fd864d9afe2b)

 I am running an rsync in the background and then doing a simple ls -la
 so the log is large.

 I am guessing this is the problem. The file is there and if I list the
 directory again it shows up properly.

 2015-09-28 16:34:21.548631 7f372effd700  3 client.28239198 ll_lookup
 0x7f370d1b1c50 data.2015-08-23_00-00-00.csv.bz2
 2015-09-28 16:34:21.548635 7f372effd700 10 client.28239198 _lookup
 concluded ENOENT locally for 19d72a1.head(ref=4 ll_ref=5 cap_refs={}
 open={} mode=42775 size=0/0 mtime=2015-09-28 05:57:57.259306
 caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE parents=0x7f3732ff97c0 0x7f370d1b1c50)
 dn 'data.2015-08-23_00-00-00.csv.bz2'


 [image: Selection_034.png]

 It seems to show up more if multiple things are access the ceph mount,
 just my observations.

 Best,
 Scott

 On Tue, Mar 3, 2015 at 3:05 PM Scottix  wrote:

> Ya we are not at 0.87.1 yet, possibly tomorrow. I'll let you know if
> it still reports the same.
>
> Thanks John,
> --Scottie
>
>
> On Tue, Mar 3, 2015 at 2:57 PM John Spray 
> wrote:
>
>> On 03/03/2015 22:35, Scottix wrote:
>> > I was testing a little bit more and decided to run the
>> cephfs-journal-tool
>> >
>> > I ran across some errors
>> >
>> > $ cephfs-journal-tool journal inspect
>> > 2015-03-03 14:18:54.453981 7f8e29f86780 -1 Bad entry start ptr
>> > (0x2aebf6) at 0x2aeb32279b
>> > 2015-03-03 14:18:54.539060 7f8e29f86780 -1 Bad entry start ptr
>> > (0x2aeb000733) at 0x2aeb322dd8
>> > 2015-03-03 14:18:54.584539 7f8e29f86780 -1 Bad entry start ptr
>> > (0x2aeb000d70) at 0x2aeb323415
>> > 2015-03-03 14:18:54.669991 7f8e29f86780 -1 Bad entry start ptr
>> > (0x2aeb0013ad) at 0x2aeb323a52
>> > 2015-03-03 14:18:54.707724 7f8e29f86780 -1 Bad entry start ptr
>> > (0x2aeb0019ea) at 0x2aeb32408f
>> > Overall journal integrity: DAMAGED
>>
>> I expect this is http://tracker.ceph.com/issues/9977, which is fixed
>> in
>> master.
>>
>> You are in *very* bleeding edge territory here, and I'd suggest using
>> the latest development release if you want to experiment with the
>> latest
>> CephFS tooling.
>>
>> Cheers,
>> John
>>
>
>>>
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS Attributes Question Marks

2015-09-30 Thread Scottix 
OpenSuse 12.1

3.1.10-1.29-desktop

On Wed, Sep 30, 2015, 5:34 AM Yan, Zheng  wrote:

> On Tue, Sep 29, 2015 at 9:51 PM, Scottix  wrote:
>
>> I'm positive the client I sent you the log is 94. We do have one client
>> still on 87.
>>
> which version of kernel are you using? I found a kernel bug which can
> cause this issue in 4.1 and later kernels.
>
> Regards
> Yan, Zheng
>
>
>
>>
>> On Tue, Sep 29, 2015, 6:42 AM John Spray  wrote:
>>
>>>
>>> Hmm, so apparently a similar bug was fixed in 0.87: Scott, can you
>>> confirm that your *clients* were 0.94 (not just the servers)?
>>>
>>> Thanks,
>>> John
>>>
>>> On Tue, Sep 29, 2015 at 11:56 AM, John Spray  wrote:
>>>
 Ah, this is a nice clear log!

 I've described the bug here:
 http://tracker.ceph.com/issues/13271

 In the short term, you may be able to mitigate this by increasing
 client_cache_size (on the client) if your RAM allows it.

 John

 On Tue, Sep 29, 2015 at 12:58 AM, Scottix  wrote:

> I know this is an old one but I got a log in ceph-fuse for it.
> I got this on OpenSuse 12.1
> 3.1.10-1.29-desktop
>
> Using ceph-fuse
> ceph version 0.94.3 (95cefea9fd9ab740263bf8bb4796fd864d9afe2b)
>
> I am running an rsync in the background and then doing a simple ls -la
> so the log is large.
>
> I am guessing this is the problem. The file is there and if I list the
> directory again it shows up properly.
>
> 2015-09-28 16:34:21.548631 7f372effd700  3 client.28239198 ll_lookup
> 0x7f370d1b1c50 data.2015-08-23_00-00-00.csv.bz2
> 2015-09-28 16:34:21.548635 7f372effd700 10 client.28239198 _lookup
> concluded ENOENT locally for 19d72a1.head(ref=4 ll_ref=5 cap_refs={}
> open={} mode=42775 size=0/0 mtime=2015-09-28 05:57:57.259306
> caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE parents=0x7f3732ff97c0 
> 0x7f370d1b1c50)
> dn 'data.2015-08-23_00-00-00.csv.bz2'
>
>
> [image: Selection_034.png]
>
> It seems to show up more if multiple things are access the ceph mount,
> just my observations.
>
> Best,
> Scott
>
> On Tue, Mar 3, 2015 at 3:05 PM Scottix  wrote:
>
>> Ya we are not at 0.87.1 yet, possibly tomorrow. I'll let you know if
>> it still reports the same.
>>
>> Thanks John,
>> --Scottie
>>
>>
>> On Tue, Mar 3, 2015 at 2:57 PM John Spray 
>> wrote:
>>
>>> On 03/03/2015 22:35, Scottix wrote:
>>> > I was testing a little bit more and decided to run the
>>> cephfs-journal-tool
>>> >
>>> > I ran across some errors
>>> >
>>> > $ cephfs-journal-tool journal inspect
>>> > 2015-03-03 14:18:54.453981 7f8e29f86780 -1 Bad entry start ptr
>>> > (0x2aebf6) at 0x2aeb32279b
>>> > 2015-03-03 14:18:54.539060 7f8e29f86780 -1 Bad entry start ptr
>>> > (0x2aeb000733) at 0x2aeb322dd8
>>> > 2015-03-03 14:18:54.584539 7f8e29f86780 -1 Bad entry start ptr
>>> > (0x2aeb000d70) at 0x2aeb323415
>>> > 2015-03-03 14:18:54.669991 7f8e29f86780 -1 Bad entry start ptr
>>> > (0x2aeb0013ad) at 0x2aeb323a52
>>> > 2015-03-03 14:18:54.707724 7f8e29f86780 -1 Bad entry start ptr
>>> > (0x2aeb0019ea) at 0x2aeb32408f
>>> > Overall journal integrity: DAMAGED
>>>
>>> I expect this is http://tracker.ceph.com/issues/9977, which is
>>> fixed in
>>> master.
>>>
>>> You are in *very* bleeding edge territory here, and I'd suggest using
>>> the latest development release if you want to experiment with the
>>> latest
>>> CephFS tooling.
>>>
>>> Cheers,
>>> John
>>>
>>

>>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS Attributes Question Marks

2015-09-29 Thread John Spray 
Hmm, so apparently a similar bug was fixed in 0.87: Scott, can you confirm
that your *clients* were 0.94 (not just the servers)?

Thanks,
John

On Tue, Sep 29, 2015 at 11:56 AM, John Spray  wrote:

> Ah, this is a nice clear log!
>
> I've described the bug here:
> http://tracker.ceph.com/issues/13271
>
> In the short term, you may be able to mitigate this by increasing
> client_cache_size (on the client) if your RAM allows it.
>
> John
>
> On Tue, Sep 29, 2015 at 12:58 AM, Scottix  wrote:
>
>> I know this is an old one but I got a log in ceph-fuse for it.
>> I got this on OpenSuse 12.1
>> 3.1.10-1.29-desktop
>>
>> Using ceph-fuse
>> ceph version 0.94.3 (95cefea9fd9ab740263bf8bb4796fd864d9afe2b)
>>
>> I am running an rsync in the background and then doing a simple ls -la so
>> the log is large.
>>
>> I am guessing this is the problem. The file is there and if I list the
>> directory again it shows up properly.
>>
>> 2015-09-28 16:34:21.548631 7f372effd700  3 client.28239198 ll_lookup
>> 0x7f370d1b1c50 data.2015-08-23_00-00-00.csv.bz2
>> 2015-09-28 16:34:21.548635 7f372effd700 10 client.28239198 _lookup
>> concluded ENOENT locally for 19d72a1.head(ref=4 ll_ref=5 cap_refs={}
>> open={} mode=42775 size=0/0 mtime=2015-09-28 05:57:57.259306
>> caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE parents=0x7f3732ff97c0 0x7f370d1b1c50)
>> dn 'data.2015-08-23_00-00-00.csv.bz2'
>>
>>
>> [image: Selection_034.png]
>>
>> It seems to show up more if multiple things are access the ceph mount,
>> just my observations.
>>
>> Best,
>> Scott
>>
>> On Tue, Mar 3, 2015 at 3:05 PM Scottix  wrote:
>>
>>> Ya we are not at 0.87.1 yet, possibly tomorrow. I'll let you know if it
>>> still reports the same.
>>>
>>> Thanks John,
>>> --Scottie
>>>
>>>
>>> On Tue, Mar 3, 2015 at 2:57 PM John Spray  wrote:
>>>
 On 03/03/2015 22:35, Scottix wrote:
 > I was testing a little bit more and decided to run the
 cephfs-journal-tool
 >
 > I ran across some errors
 >
 > $ cephfs-journal-tool journal inspect
 > 2015-03-03 14:18:54.453981 7f8e29f86780 -1 Bad entry start ptr
 > (0x2aebf6) at 0x2aeb32279b
 > 2015-03-03 14:18:54.539060 7f8e29f86780 -1 Bad entry start ptr
 > (0x2aeb000733) at 0x2aeb322dd8
 > 2015-03-03 14:18:54.584539 7f8e29f86780 -1 Bad entry start ptr
 > (0x2aeb000d70) at 0x2aeb323415
 > 2015-03-03 14:18:54.669991 7f8e29f86780 -1 Bad entry start ptr
 > (0x2aeb0013ad) at 0x2aeb323a52
 > 2015-03-03 14:18:54.707724 7f8e29f86780 -1 Bad entry start ptr
 > (0x2aeb0019ea) at 0x2aeb32408f
 > Overall journal integrity: DAMAGED

 I expect this is http://tracker.ceph.com/issues/9977, which is fixed in
 master.

 You are in *very* bleeding edge territory here, and I'd suggest using
 the latest development release if you want to experiment with the latest
 CephFS tooling.

 Cheers,
 John

>>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS Attributes Question Marks

2015-09-29 Thread John Spray 
Ah, this is a nice clear log!

I've described the bug here:
http://tracker.ceph.com/issues/13271

In the short term, you may be able to mitigate this by increasing
client_cache_size (on the client) if your RAM allows it.

John

On Tue, Sep 29, 2015 at 12:58 AM, Scottix  wrote:

> I know this is an old one but I got a log in ceph-fuse for it.
> I got this on OpenSuse 12.1
> 3.1.10-1.29-desktop
>
> Using ceph-fuse
> ceph version 0.94.3 (95cefea9fd9ab740263bf8bb4796fd864d9afe2b)
>
> I am running an rsync in the background and then doing a simple ls -la so
> the log is large.
>
> I am guessing this is the problem. The file is there and if I list the
> directory again it shows up properly.
>
> 2015-09-28 16:34:21.548631 7f372effd700  3 client.28239198 ll_lookup
> 0x7f370d1b1c50 data.2015-08-23_00-00-00.csv.bz2
> 2015-09-28 16:34:21.548635 7f372effd700 10 client.28239198 _lookup
> concluded ENOENT locally for 19d72a1.head(ref=4 ll_ref=5 cap_refs={}
> open={} mode=42775 size=0/0 mtime=2015-09-28 05:57:57.259306
> caps=pAsLsXsFs(0=pAsLsXsFs) COMPLETE parents=0x7f3732ff97c0 0x7f370d1b1c50)
> dn 'data.2015-08-23_00-00-00.csv.bz2'
>
>
> [image: Selection_034.png]
>
> It seems to show up more if multiple things are access the ceph mount,
> just my observations.
>
> Best,
> Scott
>
> On Tue, Mar 3, 2015 at 3:05 PM Scottix  wrote:
>
>> Ya we are not at 0.87.1 yet, possibly tomorrow. I'll let you know if it
>> still reports the same.
>>
>> Thanks John,
>> --Scottie
>>
>>
>> On Tue, Mar 3, 2015 at 2:57 PM John Spray  wrote:
>>
>>> On 03/03/2015 22:35, Scottix wrote:
>>> > I was testing a little bit more and decided to run the
>>> cephfs-journal-tool
>>> >
>>> > I ran across some errors
>>> >
>>> > $ cephfs-journal-tool journal inspect
>>> > 2015-03-03 14:18:54.453981 7f8e29f86780 -1 Bad entry start ptr
>>> > (0x2aebf6) at 0x2aeb32279b
>>> > 2015-03-03 14:18:54.539060 7f8e29f86780 -1 Bad entry start ptr
>>> > (0x2aeb000733) at 0x2aeb322dd8
>>> > 2015-03-03 14:18:54.584539 7f8e29f86780 -1 Bad entry start ptr
>>> > (0x2aeb000d70) at 0x2aeb323415
>>> > 2015-03-03 14:18:54.669991 7f8e29f86780 -1 Bad entry start ptr
>>> > (0x2aeb0013ad) at 0x2aeb323a52
>>> > 2015-03-03 14:18:54.707724 7f8e29f86780 -1 Bad entry start ptr
>>> > (0x2aeb0019ea) at 0x2aeb32408f
>>> > Overall journal integrity: DAMAGED
>>>
>>> I expect this is http://tracker.ceph.com/issues/9977, which is fixed in
>>> master.
>>>
>>> You are in *very* bleeding edge territory here, and I'd suggest using
>>> the latest development release if you want to experiment with the latest
>>> CephFS tooling.
>>>
>>> Cheers,
>>> John
>>>
>>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS Attributes Question Marks

2015-03-03 Thread Scottix 
I did a bit more testing.
1. I tried on a newer kernel and was not able to recreate the problem,
maybe it is that kernel bug you mentioned. Although its not an exact
replica of the load.
2. I haven't tried the debug yet since I have to wait for the right moment.

One thing I realized and maybe it is not an issue is we are using a symlink
to a folder in the ceph mount.
ceph-fuse on /mnt/ceph type fuse.ceph-fuse
(rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other)
lrwxrwxrwx 1 root   root   metadata - /mnt/ceph/DataCenter/metadata
Not sure if that would create any issues.

Anyway we are going to update the machine soon so, I can report if we keep
having the issue.

Thanks for your support,
Scott


On Mon, Mar 2, 2015 at 4:07 PM Scottix scot...@gmail.com wrote:

 I'll try the following things and report back to you.

 1. I can get a new kernel on another machine and mount to the CephFS and
 see if I get the following errors.
 2. I'll run the debug and see if anything comes up.

 I'll report back to you when I can do these things.

 Thanks,
 Scottie

 On Mon, Mar 2, 2015 at 4:04 PM Gregory Farnum g...@gregs42.com wrote:

 I bet it's that permission issue combined with a minor bug in FUSE on
 that kernel, or maybe in the ceph-fuse code (but I've not seen it
 reported before, so I kind of doubt it). If you run ceph-fuse with
 debug client = 20 it will output (a whole lot of) logging to the
 client's log file and you could see what requests are getting
 processed by the Ceph code and how it's responding. That might let you
 narrow things down. It's certainly not any kind of timeout.
 -Greg

 On Mon, Mar 2, 2015 at 3:57 PM, Scottix scot...@gmail.com wrote:
  3 Ceph servers on Ubuntu 12.04.5 - kernel 3.13.0-29-generic
 
  We have an old server that we compiled the ceph-fuse client on
  Suse11.4 - kernel 2.6.37.6-0.11
  This is the only mount we have right now.
 
  We don't have any problems reading the files and the directory shows
 full
  775 permissions and doing a second ls fixes the problem.
 
  On Mon, Mar 2, 2015 at 3:51 PM Bill Sanders billysand...@gmail.com
 wrote:
 
  Forgive me if this is unhelpful, but could it be something to do with
  permissions of the directory and not Ceph at all?
 
  http://superuser.com/a/528467
 
  Bill
 
  On Mon, Mar 2, 2015 at 3:47 PM, Gregory Farnum g...@gregs42.com
 wrote:
 
  On Mon, Mar 2, 2015 at 3:39 PM, Scottix scot...@gmail.com wrote:
   We have a file system running CephFS and for a while we had this
 issue
   when
   doing an ls -la we get question marks in the response.
  
   -rw-r--r-- 1 wwwrun root14761 Feb  9 16:06
   data.2015-02-08_00-00-00.csv.bz2
   -? ? ?  ?   ??
   data.2015-02-09_00-00-00.csv.bz2
  
   If we do another directory listing it show up fine.
  
   -rw-r--r-- 1 wwwrun root14761 Feb  9 16:06
   data.2015-02-08_00-00-00.csv.bz2
   -rw-r--r-- 1 wwwrun root13675 Feb 10 15:21
   data.2015-02-09_00-00-00.csv.bz2
  
   It hasn't been a problem but just wanted to see if this is an issue,
   could
   the attributes be timing out? We do have a lot of files in the
   filesystem so
   that could be a possible bottleneck.
 
  Huh, that's not something I've seen before. Are the systems you're
  doing this on the same? What distro and kernel version? Is it reliably
  one of them showing the question marks, or does it jump between
  systems?
  -Greg
 
  
   We are using the ceph-fuse mount.
   ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
   We are planning to do the update soon to 87.1
  
   Thanks
   Scottie
  
  
   ___
   ceph-users mailing list
   ceph-users@lists.ceph.com
   http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
  
  ___
  ceph-users mailing list
  ceph-users@lists.ceph.com
  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
 


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS Attributes Question Marks

2015-03-03 Thread Scottix 
Ya we are not at 0.87.1 yet, possibly tomorrow. I'll let you know if it
still reports the same.

Thanks John,
--Scottie


On Tue, Mar 3, 2015 at 2:57 PM John Spray john.sp...@redhat.com wrote:

 On 03/03/2015 22:35, Scottix wrote:
  I was testing a little bit more and decided to run the
 cephfs-journal-tool
 
  I ran across some errors
 
  $ cephfs-journal-tool journal inspect
  2015-03-03 14:18:54.453981 7f8e29f86780 -1 Bad entry start ptr
  (0x2aebf6) at 0x2aeb32279b
  2015-03-03 14:18:54.539060 7f8e29f86780 -1 Bad entry start ptr
  (0x2aeb000733) at 0x2aeb322dd8
  2015-03-03 14:18:54.584539 7f8e29f86780 -1 Bad entry start ptr
  (0x2aeb000d70) at 0x2aeb323415
  2015-03-03 14:18:54.669991 7f8e29f86780 -1 Bad entry start ptr
  (0x2aeb0013ad) at 0x2aeb323a52
  2015-03-03 14:18:54.707724 7f8e29f86780 -1 Bad entry start ptr
  (0x2aeb0019ea) at 0x2aeb32408f
  Overall journal integrity: DAMAGED

 I expect this is http://tracker.ceph.com/issues/9977, which is fixed in
 master.

 You are in *very* bleeding edge territory here, and I'd suggest using
 the latest development release if you want to experiment with the latest
 CephFS tooling.

 Cheers,
 John

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS Attributes Question Marks

2015-03-03 Thread Scottix 
I was testing a little bit more and decided to run the cephfs-journal-tool

I ran across some errors

$ cephfs-journal-tool journal inspect
2015-03-03 14:18:54.453981 7f8e29f86780 -1 Bad entry start ptr
(0x2aebf6) at 0x2aeb32279b
2015-03-03 14:18:54.539060 7f8e29f86780 -1 Bad entry start ptr
(0x2aeb000733) at 0x2aeb322dd8
2015-03-03 14:18:54.584539 7f8e29f86780 -1 Bad entry start ptr
(0x2aeb000d70) at 0x2aeb323415
2015-03-03 14:18:54.669991 7f8e29f86780 -1 Bad entry start ptr
(0x2aeb0013ad) at 0x2aeb323a52
2015-03-03 14:18:54.707724 7f8e29f86780 -1 Bad entry start ptr
(0x2aeb0019ea) at 0x2aeb32408f
Overall journal integrity: DAMAGED
Corrupt regions:
  0x2aeb3226a5-2aeb32279b
  0x2aeb32279b-2aeb322dd8
  0x2aeb322dd8-2aeb323415
  0x2aeb323415-2aeb323a52
  0x2aeb323a52-2aeb32408f
  0x2aeb32408f-2aeb3246cc

$ cephfs-journal-tool header get
{ magic: ceph fs volume v011,
  write_pos: 184430420380,
  expire_pos: 184389995327,
  trimmed_pos: 184389992448,
  stream_format: 1,
  layout: { stripe_unit: 4194304,
  stripe_count: 4194304,
  object_size: 4194304,
  cas_hash: 4194304,
  object_stripe_unit: 4194304,
  pg_pool: 4194304}}

$ cephfs-journal-tool event get summary
2015-03-03 14:32:50.102863 7f47c3006780 -1 Bad entry start ptr
(0x2aee8000e6) at 0x2aee800c25
2015-03-03 14:32:50.242576 7f47c3006780 -1 Bad entry start ptr
(0x2aee800b3f) at 0x2aee80167e
2015-03-03 14:32:50.486354 7f47c3006780 -1 Bad entry start ptr
(0x2aee800e4f) at 0x2aee80198e
2015-03-03 14:32:50.577443 7f47c3006780 -1 Bad entry start ptr
(0x2aee801f65) at 0x2aee802aa4
Events by type:
no output here


On Tue, Mar 3, 2015 at 12:01 PM Scottix scot...@gmail.com wrote:

 I did a bit more testing.
 1. I tried on a newer kernel and was not able to recreate the problem,
 maybe it is that kernel bug you mentioned. Although its not an exact
 replica of the load.
 2. I haven't tried the debug yet since I have to wait for the right moment.

 One thing I realized and maybe it is not an issue is we are using a
 symlink to a folder in the ceph mount.
 ceph-fuse on /mnt/ceph type fuse.ceph-fuse
 (rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other)
 lrwxrwxrwx 1 root   root   metadata - /mnt/ceph/DataCenter/metadata
 Not sure if that would create any issues.

 Anyway we are going to update the machine soon so, I can report if we keep
 having the issue.

 Thanks for your support,
 Scott


 On Mon, Mar 2, 2015 at 4:07 PM Scottix scot...@gmail.com wrote:

 I'll try the following things and report back to you.

 1. I can get a new kernel on another machine and mount to the CephFS and
 see if I get the following errors.
 2. I'll run the debug and see if anything comes up.

 I'll report back to you when I can do these things.

 Thanks,
 Scottie

 On Mon, Mar 2, 2015 at 4:04 PM Gregory Farnum g...@gregs42.com wrote:

 I bet it's that permission issue combined with a minor bug in FUSE on
 that kernel, or maybe in the ceph-fuse code (but I've not seen it
 reported before, so I kind of doubt it). If you run ceph-fuse with
 debug client = 20 it will output (a whole lot of) logging to the
 client's log file and you could see what requests are getting
 processed by the Ceph code and how it's responding. That might let you
 narrow things down. It's certainly not any kind of timeout.
 -Greg

 On Mon, Mar 2, 2015 at 3:57 PM, Scottix scot...@gmail.com wrote:
  3 Ceph servers on Ubuntu 12.04.5 - kernel 3.13.0-29-generic
 
  We have an old server that we compiled the ceph-fuse client on
  Suse11.4 - kernel 2.6.37.6-0.11
  This is the only mount we have right now.
 
  We don't have any problems reading the files and the directory shows
 full
  775 permissions and doing a second ls fixes the problem.
 
  On Mon, Mar 2, 2015 at 3:51 PM Bill Sanders billysand...@gmail.com
 wrote:
 
  Forgive me if this is unhelpful, but could it be something to do with
  permissions of the directory and not Ceph at all?
 
  http://superuser.com/a/528467
 
  Bill
 
  On Mon, Mar 2, 2015 at 3:47 PM, Gregory Farnum g...@gregs42.com
 wrote:
 
  On Mon, Mar 2, 2015 at 3:39 PM, Scottix scot...@gmail.com wrote:
   We have a file system running CephFS and for a while we had this
 issue
   when
   doing an ls -la we get question marks in the response.
  
   -rw-r--r-- 1 wwwrun root14761 Feb  9 16:06
   data.2015-02-08_00-00-00.csv.bz2
   -? ? ?  ?   ??
   data.2015-02-09_00-00-00.csv.bz2
  
   If we do another directory listing it show up fine.
  
   -rw-r--r-- 1 wwwrun root14761 Feb  9 16:06
   data.2015-02-08_00-00-00.csv.bz2
   -rw-r--r-- 1 wwwrun root13675 Feb 10 15:21
   data.2015-02-09_00-00-00.csv.bz2
  
   It hasn't been a problem but just wanted to see if this is an
 issue,
   could
   the attributes be timing out? We do have a lot of files in the
   filesystem so
   that could be a possible bottleneck.
 
  Huh, that's not something I've seen before. Are the systems you're
  doing this on the same? What

Re: [ceph-users] CephFS Attributes Question Marks

2015-03-03 Thread John Spray 

On 03/03/2015 22:35, Scottix wrote:

I was testing a little bit more and decided to run the cephfs-journal-tool

I ran across some errors

$ cephfs-journal-tool journal inspect
2015-03-03 14:18:54.453981 7f8e29f86780 -1 Bad entry start ptr 
(0x2aebf6) at 0x2aeb32279b
2015-03-03 14:18:54.539060 7f8e29f86780 -1 Bad entry start ptr 
(0x2aeb000733) at 0x2aeb322dd8
2015-03-03 14:18:54.584539 7f8e29f86780 -1 Bad entry start ptr 
(0x2aeb000d70) at 0x2aeb323415
2015-03-03 14:18:54.669991 7f8e29f86780 -1 Bad entry start ptr 
(0x2aeb0013ad) at 0x2aeb323a52
2015-03-03 14:18:54.707724 7f8e29f86780 -1 Bad entry start ptr 
(0x2aeb0019ea) at 0x2aeb32408f

Overall journal integrity: DAMAGED


I expect this is http://tracker.ceph.com/issues/9977, which is fixed in 
master.


You are in *very* bleeding edge territory here, and I'd suggest using 
the latest development release if you want to experiment with the latest 
CephFS tooling.


Cheers,
John
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS Attributes Question Marks

2015-03-03 Thread John Spray 

On 03/03/2015 22:57, John Spray wrote:

On 03/03/2015 22:35, Scottix wrote:
I was testing a little bit more and decided to run the 
cephfs-journal-tool


I ran across some errors

$ cephfs-journal-tool journal inspect
2015-03-03 14:18:54.453981 7f8e29f86780 -1 Bad entry start ptr 
(0x2aebf6) at 0x2aeb32279b
2015-03-03 14:18:54.539060 7f8e29f86780 -1 Bad entry start ptr 
(0x2aeb000733) at 0x2aeb322dd8
2015-03-03 14:18:54.584539 7f8e29f86780 -1 Bad entry start ptr 
(0x2aeb000d70) at 0x2aeb323415
2015-03-03 14:18:54.669991 7f8e29f86780 -1 Bad entry start ptr 
(0x2aeb0013ad) at 0x2aeb323a52
2015-03-03 14:18:54.707724 7f8e29f86780 -1 Bad entry start ptr 
(0x2aeb0019ea) at 0x2aeb32408f

Overall journal integrity: DAMAGED


I expect this is http://tracker.ceph.com/issues/9977, which is fixed 
in master.


You are in *very* bleeding edge territory here, and I'd suggest using 
the latest development release if you want to experiment with the 
latest CephFS tooling.
...although at the risk of contradicting myself, I now notice that this 
particular bugfix is one that we did backport for 0.87.1


John

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS Attributes Question Marks

2015-03-02 Thread Bill Sanders 
Forgive me if this is unhelpful, but could it be something to do with
permissions of the directory and not Ceph at all?

http://superuser.com/a/528467

Bill

On Mon, Mar 2, 2015 at 3:47 PM, Gregory Farnum g...@gregs42.com wrote:

 On Mon, Mar 2, 2015 at 3:39 PM, Scottix scot...@gmail.com wrote:
  We have a file system running CephFS and for a while we had this issue
 when
  doing an ls -la we get question marks in the response.
 
  -rw-r--r-- 1 wwwrun root14761 Feb  9 16:06
  data.2015-02-08_00-00-00.csv.bz2
  -? ? ?  ?   ??
  data.2015-02-09_00-00-00.csv.bz2
 
  If we do another directory listing it show up fine.
 
  -rw-r--r-- 1 wwwrun root14761 Feb  9 16:06
  data.2015-02-08_00-00-00.csv.bz2
  -rw-r--r-- 1 wwwrun root13675 Feb 10 15:21
  data.2015-02-09_00-00-00.csv.bz2
 
  It hasn't been a problem but just wanted to see if this is an issue,
 could
  the attributes be timing out? We do have a lot of files in the
 filesystem so
  that could be a possible bottleneck.

 Huh, that's not something I've seen before. Are the systems you're
 doing this on the same? What distro and kernel version? Is it reliably
 one of them showing the question marks, or does it jump between
 systems?
 -Greg

 
  We are using the ceph-fuse mount.
  ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
  We are planning to do the update soon to 87.1
 
  Thanks
  Scottie
 
 
  ___
  ceph-users mailing list
  ceph-users@lists.ceph.com
  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS Attributes Question Marks

2015-03-02 Thread Gregory Farnum 
I bet it's that permission issue combined with a minor bug in FUSE on
that kernel, or maybe in the ceph-fuse code (but I've not seen it
reported before, so I kind of doubt it). If you run ceph-fuse with
debug client = 20 it will output (a whole lot of) logging to the
client's log file and you could see what requests are getting
processed by the Ceph code and how it's responding. That might let you
narrow things down. It's certainly not any kind of timeout.
-Greg

On Mon, Mar 2, 2015 at 3:57 PM, Scottix scot...@gmail.com wrote:
 3 Ceph servers on Ubuntu 12.04.5 - kernel 3.13.0-29-generic

 We have an old server that we compiled the ceph-fuse client on
 Suse11.4 - kernel 2.6.37.6-0.11
 This is the only mount we have right now.

 We don't have any problems reading the files and the directory shows full
 775 permissions and doing a second ls fixes the problem.

 On Mon, Mar 2, 2015 at 3:51 PM Bill Sanders billysand...@gmail.com wrote:

 Forgive me if this is unhelpful, but could it be something to do with
 permissions of the directory and not Ceph at all?

 http://superuser.com/a/528467

 Bill

 On Mon, Mar 2, 2015 at 3:47 PM, Gregory Farnum g...@gregs42.com wrote:

 On Mon, Mar 2, 2015 at 3:39 PM, Scottix scot...@gmail.com wrote:
  We have a file system running CephFS and for a while we had this issue
  when
  doing an ls -la we get question marks in the response.
 
  -rw-r--r-- 1 wwwrun root14761 Feb  9 16:06
  data.2015-02-08_00-00-00.csv.bz2
  -? ? ?  ?   ??
  data.2015-02-09_00-00-00.csv.bz2
 
  If we do another directory listing it show up fine.
 
  -rw-r--r-- 1 wwwrun root14761 Feb  9 16:06
  data.2015-02-08_00-00-00.csv.bz2
  -rw-r--r-- 1 wwwrun root13675 Feb 10 15:21
  data.2015-02-09_00-00-00.csv.bz2
 
  It hasn't been a problem but just wanted to see if this is an issue,
  could
  the attributes be timing out? We do have a lot of files in the
  filesystem so
  that could be a possible bottleneck.

 Huh, that's not something I've seen before. Are the systems you're
 doing this on the same? What distro and kernel version? Is it reliably
 one of them showing the question marks, or does it jump between
 systems?
 -Greg

 
  We are using the ceph-fuse mount.
  ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
  We are planning to do the update soon to 87.1
 
  Thanks
  Scottie
 
 
  ___
  ceph-users mailing list
  ceph-users@lists.ceph.com
  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS Attributes Question Marks

2015-03-02 Thread Gregory Farnum 
On Mon, Mar 2, 2015 at 3:39 PM, Scottix scot...@gmail.com wrote:
 We have a file system running CephFS and for a while we had this issue when
 doing an ls -la we get question marks in the response.

 -rw-r--r-- 1 wwwrun root14761 Feb  9 16:06
 data.2015-02-08_00-00-00.csv.bz2
 -? ? ?  ?   ??
 data.2015-02-09_00-00-00.csv.bz2

 If we do another directory listing it show up fine.

 -rw-r--r-- 1 wwwrun root14761 Feb  9 16:06
 data.2015-02-08_00-00-00.csv.bz2
 -rw-r--r-- 1 wwwrun root13675 Feb 10 15:21
 data.2015-02-09_00-00-00.csv.bz2

 It hasn't been a problem but just wanted to see if this is an issue, could
 the attributes be timing out? We do have a lot of files in the filesystem so
 that could be a possible bottleneck.

Huh, that's not something I've seen before. Are the systems you're
doing this on the same? What distro and kernel version? Is it reliably
one of them showing the question marks, or does it jump between
systems?
-Greg


 We are using the ceph-fuse mount.
 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
 We are planning to do the update soon to 87.1

 Thanks
 Scottie


 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS Attributes Question Marks

2015-03-02 Thread Scottix 
3 Ceph servers on Ubuntu 12.04.5 - kernel 3.13.0-29-generic

We have an old server that we compiled the ceph-fuse client on
Suse11.4 - kernel 2.6.37.6-0.11
This is the only mount we have right now.

We don't have any problems reading the files and the directory shows full
775 permissions and doing a second ls fixes the problem.

On Mon, Mar 2, 2015 at 3:51 PM Bill Sanders billysand...@gmail.com wrote:

 Forgive me if this is unhelpful, but could it be something to do with
 permissions of the directory and not Ceph at all?

 http://superuser.com/a/528467

 Bill

 On Mon, Mar 2, 2015 at 3:47 PM, Gregory Farnum g...@gregs42.com wrote:

 On Mon, Mar 2, 2015 at 3:39 PM, Scottix scot...@gmail.com wrote:
  We have a file system running CephFS and for a while we had this issue
 when
  doing an ls -la we get question marks in the response.
 
  -rw-r--r-- 1 wwwrun root14761 Feb  9 16:06
  data.2015-02-08_00-00-00.csv.bz2
  -? ? ?  ?   ??
  data.2015-02-09_00-00-00.csv.bz2
 
  If we do another directory listing it show up fine.
 
  -rw-r--r-- 1 wwwrun root14761 Feb  9 16:06
  data.2015-02-08_00-00-00.csv.bz2
  -rw-r--r-- 1 wwwrun root13675 Feb 10 15:21
  data.2015-02-09_00-00-00.csv.bz2
 
  It hasn't been a problem but just wanted to see if this is an issue,
 could
  the attributes be timing out? We do have a lot of files in the
 filesystem so
  that could be a possible bottleneck.

 Huh, that's not something I've seen before. Are the systems you're
 doing this on the same? What distro and kernel version? Is it reliably
 one of them showing the question marks, or does it jump between
 systems?
 -Greg

 
  We are using the ceph-fuse mount.
  ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
  We are planning to do the update soon to 87.1
 
  Thanks
  Scottie
 
 
  ___
  ceph-users mailing list
  ceph-users@lists.ceph.com
  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS Attributes Question Marks

2015-03-02 Thread Scottix 
I'll try the following things and report back to you.

1. I can get a new kernel on another machine and mount to the CephFS and
see if I get the following errors.
2. I'll run the debug and see if anything comes up.

I'll report back to you when I can do these things.

Thanks,
Scottie

On Mon, Mar 2, 2015 at 4:04 PM Gregory Farnum g...@gregs42.com wrote:

 I bet it's that permission issue combined with a minor bug in FUSE on
 that kernel, or maybe in the ceph-fuse code (but I've not seen it
 reported before, so I kind of doubt it). If you run ceph-fuse with
 debug client = 20 it will output (a whole lot of) logging to the
 client's log file and you could see what requests are getting
 processed by the Ceph code and how it's responding. That might let you
 narrow things down. It's certainly not any kind of timeout.
 -Greg

 On Mon, Mar 2, 2015 at 3:57 PM, Scottix scot...@gmail.com wrote:
  3 Ceph servers on Ubuntu 12.04.5 - kernel 3.13.0-29-generic
 
  We have an old server that we compiled the ceph-fuse client on
  Suse11.4 - kernel 2.6.37.6-0.11
  This is the only mount we have right now.
 
  We don't have any problems reading the files and the directory shows full
  775 permissions and doing a second ls fixes the problem.
 
  On Mon, Mar 2, 2015 at 3:51 PM Bill Sanders billysand...@gmail.com
 wrote:
 
  Forgive me if this is unhelpful, but could it be something to do with
  permissions of the directory and not Ceph at all?
 
  http://superuser.com/a/528467
 
  Bill
 
  On Mon, Mar 2, 2015 at 3:47 PM, Gregory Farnum g...@gregs42.com
 wrote:
 
  On Mon, Mar 2, 2015 at 3:39 PM, Scottix scot...@gmail.com wrote:
   We have a file system running CephFS and for a while we had this
 issue
   when
   doing an ls -la we get question marks in the response.
  
   -rw-r--r-- 1 wwwrun root14761 Feb  9 16:06
   data.2015-02-08_00-00-00.csv.bz2
   -? ? ?  ?   ??
   data.2015-02-09_00-00-00.csv.bz2
  
   If we do another directory listing it show up fine.
  
   -rw-r--r-- 1 wwwrun root14761 Feb  9 16:06
   data.2015-02-08_00-00-00.csv.bz2
   -rw-r--r-- 1 wwwrun root13675 Feb 10 15:21
   data.2015-02-09_00-00-00.csv.bz2
  
   It hasn't been a problem but just wanted to see if this is an issue,
   could
   the attributes be timing out? We do have a lot of files in the
   filesystem so
   that could be a possible bottleneck.
 
  Huh, that's not something I've seen before. Are the systems you're
  doing this on the same? What distro and kernel version? Is it reliably
  one of them showing the question marks, or does it jump between
  systems?
  -Greg
 
  
   We are using the ceph-fuse mount.
   ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
   We are planning to do the update soon to 87.1
  
   Thanks
   Scottie
  
  
   ___
   ceph-users mailing list
   ceph-users@lists.ceph.com
   http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
  
  ___
  ceph-users mailing list
  ceph-users@lists.ceph.com
  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com