I was testing a little bit more and decided to run the cephfs-journal-tool
I ran across some errors
$ cephfs-journal-tool journal inspect
2015-03-03 14:18:54.453981 7f8e29f86780 -1 Bad entry start ptr
(0x2aeb0000f6) at 0x2aeb32279b
2015-03-03 14:18:54.539060 7f8e29f86780 -1 Bad entry start ptr
(0x2aeb000733) at 0x2aeb322dd8
2015-03-03 14:18:54.584539 7f8e29f86780 -1 Bad entry start ptr
(0x2aeb000d70) at 0x2aeb323415
2015-03-03 14:18:54.669991 7f8e29f86780 -1 Bad entry start ptr
(0x2aeb0013ad) at 0x2aeb323a52
2015-03-03 14:18:54.707724 7f8e29f86780 -1 Bad entry start ptr
(0x2aeb0019ea) at 0x2aeb32408f
Overall journal integrity: DAMAGED
Corrupt regions:
0x2aeb3226a5-2aeb32279b
0x2aeb32279b-2aeb322dd8
0x2aeb322dd8-2aeb323415
0x2aeb323415-2aeb323a52
0x2aeb323a52-2aeb32408f
0x2aeb32408f-2aeb3246cc
$ cephfs-journal-tool header get
{ "magic": "ceph fs volume v011",
"write_pos": 184430420380,
"expire_pos": 184389995327,
"trimmed_pos": 184389992448,
"stream_format": 1,
"layout": { "stripe_unit": 4194304,
"stripe_count": 4194304,
"object_size": 4194304,
"cas_hash": 4194304,
"object_stripe_unit": 4194304,
"pg_pool": 4194304}}
$ cephfs-journal-tool event get summary
2015-03-03 14:32:50.102863 7f47c3006780 -1 Bad entry start ptr
(0x2aee8000e6) at 0x2aee800c25
2015-03-03 14:32:50.242576 7f47c3006780 -1 Bad entry start ptr
(0x2aee800b3f) at 0x2aee80167e
2015-03-03 14:32:50.486354 7f47c3006780 -1 Bad entry start ptr
(0x2aee800e4f) at 0x2aee80198e
2015-03-03 14:32:50.577443 7f47c3006780 -1 Bad entry start ptr
(0x2aee801f65) at 0x2aee802aa4
Events by type:
<no output here>
On Tue, Mar 3, 2015 at 12:01 PM Scottix <[email protected]> wrote:
> I did a bit more testing.
> 1. I tried on a newer kernel and was not able to recreate the problem,
> maybe it is that kernel bug you mentioned. Although its not an exact
> replica of the load.
> 2. I haven't tried the debug yet since I have to wait for the right moment.
>
> One thing I realized and maybe it is not an issue is we are using a
> symlink to a folder in the ceph mount.
> ceph-fuse on /mnt/ceph type fuse.ceph-fuse
> (rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other)
> lrwxrwxrwx 1 root root metadata -> /mnt/ceph/DataCenter/metadata
> Not sure if that would create any issues.
>
> Anyway we are going to update the machine soon so, I can report if we keep
> having the issue.
>
> Thanks for your support,
> Scott
>
>
> On Mon, Mar 2, 2015 at 4:07 PM Scottix <[email protected]> wrote:
>
>> I'll try the following things and report back to you.
>>
>> 1. I can get a new kernel on another machine and mount to the CephFS and
>> see if I get the following errors.
>> 2. I'll run the debug and see if anything comes up.
>>
>> I'll report back to you when I can do these things.
>>
>> Thanks,
>> Scottie
>>
>> On Mon, Mar 2, 2015 at 4:04 PM Gregory Farnum <[email protected]> wrote:
>>
>>> I bet it's that permission issue combined with a minor bug in FUSE on
>>> that kernel, or maybe in the ceph-fuse code (but I've not seen it
>>> reported before, so I kind of doubt it). If you run ceph-fuse with
>>> "debug client = 20" it will output (a whole lot of) logging to the
>>> client's log file and you could see what requests are getting
>>> processed by the Ceph code and how it's responding. That might let you
>>> narrow things down. It's certainly not any kind of timeout.
>>> -Greg
>>>
>>> On Mon, Mar 2, 2015 at 3:57 PM, Scottix <[email protected]> wrote:
>>> > 3 Ceph servers on Ubuntu 12.04.5 - kernel 3.13.0-29-generic
>>> >
>>> > We have an old server that we compiled the ceph-fuse client on
>>> > Suse11.4 - kernel 2.6.37.6-0.11
>>> > This is the only mount we have right now.
>>> >
>>> > We don't have any problems reading the files and the directory shows
>>> full
>>> > 775 permissions and doing a second ls fixes the problem.
>>> >
>>> > On Mon, Mar 2, 2015 at 3:51 PM Bill Sanders <[email protected]>
>>> wrote:
>>> >>
>>> >> Forgive me if this is unhelpful, but could it be something to do with
>>> >> permissions of the directory and not Ceph at all?
>>> >>
>>> >> http://superuser.com/a/528467
>>> >>
>>> >> Bill
>>> >>
>>> >> On Mon, Mar 2, 2015 at 3:47 PM, Gregory Farnum <[email protected]>
>>> wrote:
>>> >>>
>>> >>> On Mon, Mar 2, 2015 at 3:39 PM, Scottix <[email protected]> wrote:
>>> >>> > We have a file system running CephFS and for a while we had this
>>> issue
>>> >>> > when
>>> >>> > doing an ls -la we get question marks in the response.
>>> >>> >
>>> >>> > -rw-r--r-- 1 wwwrun root 14761 Feb 9 16:06
>>> >>> > data.2015-02-08_00-00-00.csv.bz2
>>> >>> > -????????? ? ? ? ? ?
>>> >>> > data.2015-02-09_00-00-00.csv.bz2
>>> >>> >
>>> >>> > If we do another directory listing it show up fine.
>>> >>> >
>>> >>> > -rw-r--r-- 1 wwwrun root 14761 Feb 9 16:06
>>> >>> > data.2015-02-08_00-00-00.csv.bz2
>>> >>> > -rw-r--r-- 1 wwwrun root 13675 Feb 10 15:21
>>> >>> > data.2015-02-09_00-00-00.csv.bz2
>>> >>> >
>>> >>> > It hasn't been a problem but just wanted to see if this is an
>>> issue,
>>> >>> > could
>>> >>> > the attributes be timing out? We do have a lot of files in the
>>> >>> > filesystem so
>>> >>> > that could be a possible bottleneck.
>>> >>>
>>> >>> Huh, that's not something I've seen before. Are the systems you're
>>> >>> doing this on the same? What distro and kernel version? Is it
>>> reliably
>>> >>> one of them showing the question marks, or does it jump between
>>> >>> systems?
>>> >>> -Greg
>>> >>>
>>> >>> >
>>> >>> > We are using the ceph-fuse mount.
>>> >>> > ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
>>> >>> > We are planning to do the update soon to 87.1
>>> >>> >
>>> >>> > Thanks
>>> >>> > Scottie
>>> >>> >
>>> >>> >
>>> >>> > _______________________________________________
>>> >>> > ceph-users mailing list
>>> >>> > [email protected]
>>> >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> >>> >
>>> >>> _______________________________________________
>>> >>> ceph-users mailing list
>>> >>> [email protected]
>>> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> >>
>>> >>
>>> >
>>>
>>
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
