Re: [ceph-users] Corrupted files on CephFS since Luminous upgrade

[Yan, Zheng]

On Tue, Feb 27, 2018 at 2:29 PM, Jan Pekař - Imatic  wrote:
> I think I hit the same issue.
> I have corrupted data on cephfs and I don't remember the same issue before
> Luminous (i did the same tests before).
>
> It is on my test 1 node cluster with lower memory then recommended (so
> server is swapping) but it shouldn't lose data (it never did before).
> So slow requests may appear in the log like Florent B mentioned.
>
> My test is to take some bigger files (few GB) and copy it to cephfs or from
> cephfs to cephfs and stress the cluster so data copying stall for a while.
> It will resume in few seconds/minutes and everything looks ok (no error on
> copying). But copied file may be corrupted silently.
>
> I checked wiles with MD5SUM and compared some corrupted files in detail.
> There were missing some 4MB blocks of data (cephfs object size) - corrupted
> file had that block of data filled with zeroes.
>
> My idea is, that there happen something wrong when cluster is under pressure
> and client want to save the block. Client gets OK and continues with another
> block so data is lost and corrupted block is filled with zeros.
>
> I tried kernel client 4.x and ceph-fuse client with same result.
>

exact version of kernel on which you reproduced this issue?

> I'm using erasure for cephfs data pool, cache tier and my storage is
> bluestore and filestore mixed.
>
> How can I help to debug or what should I do to help to find the problem?
>
> With regards
> Jan Pekar
>
>
> On 14.12.2017 15:41, Yan, Zheng wrote:
>>
>> On Thu, Dec 14, 2017 at 8:52 PM, Florent B  wrote:
>>>
>>> On 14/12/2017 03:38, Yan, Zheng wrote:

 On Thu, Dec 14, 2017 at 12:49 AM, Florent B  wrote:
>
>
> Systems are on Debian Jessie : kernel 3.16.0-4-amd64 & libfuse
> 2.9.3-15.
>
> I don't know pattern of corruption, but according to error message in
> Dovecot, it seems to expect data to read but reach EOF.
>
> All seems fine using fuse_disable_pagecache (no more corruption, and
> performance increased : no more MDS slow requests on filelock
> requests).


 I checked ceph-fuse changes since kraken, didn't find any clue. I
 would be helpful if you can try recent version kernel.

 Regards
 Yan, Zheng
>>>
>>>
>>> Problem occurred this morning even with fuse_disable_pagecache=true.
>>>
>>> It seems to be a lock issue between imap & lmtp processes.
>>>
>>> Dovecot uses fcntl as locking method. Is there any change about it in
>>> Luminous ? I switched to flock to see if problem is still there...
>>>
>>
>> I don't remenber there is any change.
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
> --
> 
> Ing. Jan Pekař
> jan.pe...@imatic.cz | +420603811737
> 
> Imatic | Jagellonská 14 | Praha 3 | 130 00
> http://www.imatic.cz
> 
> --
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Corrupted files on CephFS since Luminous upgrade

[Yan, Zheng]

On Sat, Mar 3, 2018 at 6:17 PM, Jan Pekař - Imatic  wrote:
> On 3.3.2018 11:12, Yan, Zheng wrote:
>>
>> On Tue, Feb 27, 2018 at 2:29 PM, Jan Pekař - Imatic 
>> wrote:
>>>
>>> I think I hit the same issue.
>>> I have corrupted data on cephfs and I don't remember the same issue
>>> before
>>> Luminous (i did the same tests before).
>>>
>>> It is on my test 1 node cluster with lower memory then recommended (so
>>> server is swapping) but it shouldn't lose data (it never did before).
>>> So slow requests may appear in the log like Florent B mentioned.
>>>
>>> My test is to take some bigger files (few GB) and copy it to cephfs or
>>> from
>>> cephfs to cephfs and stress the cluster so data copying stall for a
>>> while.
>>> It will resume in few seconds/minutes and everything looks ok (no error
>>> on
>>> copying). But copied file may be corrupted silently.
>>>
>>> I checked wiles with MD5SUM and compared some corrupted files in detail.
>>> There were missing some 4MB blocks of data (cephfs object size) -
>>> corrupted
>>> file had that block of data filled with zeroes.
>>>
>>> My idea is, that there happen something wrong when cluster is under
>>> pressure
>>> and client want to save the block. Client gets OK and continues with
>>> another
>>> block so data is lost and corrupted block is filled with zeros.
>>>
>>
>> was your cluster near full when the issue happens ?
>
>
> No, I changed limits to higher near full / backfill full values, so it was
> healthy.
>

Could you please run ceph-fuse with debug_ms=1. reproduce this issue,
identify lost data and send ceph-fuse log to us.

Regards
Yan, Zheng

>
>>
>> Regards
>> Yan, Zheng
>>
>>> I tried kernel client 4.x and ceph-fuse client with same result.
>>>
>>> I'm using erasure for cephfs data pool, cache tier and my storage is
>>> bluestore and filestore mixed.
>>>
>>> How can I help to debug or what should I do to help to find the problem?
>>>
>>> With regards
>>> Jan Pekar
>>>
>>>
>>> On 14.12.2017 15:41, Yan, Zheng wrote:


 On Thu, Dec 14, 2017 at 8:52 PM, Florent B  wrote:
>
>
> On 14/12/2017 03:38, Yan, Zheng wrote:
>>
>>
>> On Thu, Dec 14, 2017 at 12:49 AM, Florent B 
>> wrote:
>>>
>>>
>>>
>>> Systems are on Debian Jessie : kernel 3.16.0-4-amd64 & libfuse
>>> 2.9.3-15.
>>>
>>> I don't know pattern of corruption, but according to error message in
>>> Dovecot, it seems to expect data to read but reach EOF.
>>>
>>> All seems fine using fuse_disable_pagecache (no more corruption, and
>>> performance increased : no more MDS slow requests on filelock
>>> requests).
>>
>>
>>
>> I checked ceph-fuse changes since kraken, didn't find any clue. I
>> would be helpful if you can try recent version kernel.
>>
>> Regards
>> Yan, Zheng
>
>
>
> Problem occurred this morning even with fuse_disable_pagecache=true.
>
> It seems to be a lock issue between imap & lmtp processes.
>
> Dovecot uses fcntl as locking method. Is there any change about it in
> Luminous ? I switched to flock to see if problem is still there...
>

 I don't remenber there is any change.
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

>>>
>>> --
>>> 
>>> Ing. Jan Pekař
>>> jan.pe...@imatic.cz | +420603811737
>>> 
>>> Imatic | Jagellonská 14 | Praha 3 | 130 00
>>> http://www.imatic.cz
>>> 
>>> --
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Corrupted files on CephFS since Luminous upgrade

[Jan Pekař - Imatic]

On 3.3.2018 11:12, Yan, Zheng wrote:

On Tue, Feb 27, 2018 at 2:29 PM, Jan Pekař - Imatic  wrote:

I think I hit the same issue.
I have corrupted data on cephfs and I don't remember the same issue before
Luminous (i did the same tests before).

It is on my test 1 node cluster with lower memory then recommended (so
server is swapping) but it shouldn't lose data (it never did before).
So slow requests may appear in the log like Florent B mentioned.

My test is to take some bigger files (few GB) and copy it to cephfs or from
cephfs to cephfs and stress the cluster so data copying stall for a while.
It will resume in few seconds/minutes and everything looks ok (no error on
copying). But copied file may be corrupted silently.

I checked wiles with MD5SUM and compared some corrupted files in detail.
There were missing some 4MB blocks of data (cephfs object size) - corrupted
file had that block of data filled with zeroes.

My idea is, that there happen something wrong when cluster is under pressure
and client want to save the block. Client gets OK and continues with another
block so data is lost and corrupted block is filled with zeros.



was your cluster near full when the issue happens ?


No, I changed limits to higher near full / backfill full values, so it 
was healthy.




Regards
Yan, Zheng


I tried kernel client 4.x and ceph-fuse client with same result.

I'm using erasure for cephfs data pool, cache tier and my storage is
bluestore and filestore mixed.

How can I help to debug or what should I do to help to find the problem?

With regards
Jan Pekar


On 14.12.2017 15:41, Yan, Zheng wrote:


On Thu, Dec 14, 2017 at 8:52 PM, Florent B  wrote:


On 14/12/2017 03:38, Yan, Zheng wrote:


On Thu, Dec 14, 2017 at 12:49 AM, Florent B  wrote:



Systems are on Debian Jessie : kernel 3.16.0-4-amd64 & libfuse
2.9.3-15.

I don't know pattern of corruption, but according to error message in
Dovecot, it seems to expect data to read but reach EOF.

All seems fine using fuse_disable_pagecache (no more corruption, and
performance increased : no more MDS slow requests on filelock
requests).



I checked ceph-fuse changes since kraken, didn't find any clue. I
would be helpful if you can try recent version kernel.

Regards
Yan, Zheng



Problem occurred this morning even with fuse_disable_pagecache=true.

It seems to be a lock issue between imap & lmtp processes.

Dovecot uses fcntl as locking method. Is there any change about it in
Luminous ? I switched to flock to see if problem is still there...



I don't remenber there is any change.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



--

Ing. Jan Pekař
jan.pe...@imatic.cz | +420603811737

Imatic | Jagellonská 14 | Praha 3 | 130 00
http://www.imatic.cz

--

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Corrupted files on CephFS since Luminous upgrade

[Yan, Zheng]

On Tue, Feb 27, 2018 at 2:29 PM, Jan Pekař - Imatic  wrote:
> I think I hit the same issue.
> I have corrupted data on cephfs and I don't remember the same issue before
> Luminous (i did the same tests before).
>
> It is on my test 1 node cluster with lower memory then recommended (so
> server is swapping) but it shouldn't lose data (it never did before).
> So slow requests may appear in the log like Florent B mentioned.
>
> My test is to take some bigger files (few GB) and copy it to cephfs or from
> cephfs to cephfs and stress the cluster so data copying stall for a while.
> It will resume in few seconds/minutes and everything looks ok (no error on
> copying). But copied file may be corrupted silently.
>
> I checked wiles with MD5SUM and compared some corrupted files in detail.
> There were missing some 4MB blocks of data (cephfs object size) - corrupted
> file had that block of data filled with zeroes.
>
> My idea is, that there happen something wrong when cluster is under pressure
> and client want to save the block. Client gets OK and continues with another
> block so data is lost and corrupted block is filled with zeros.
>

was your cluster near full when the issue happens ?

Regards
Yan, Zheng

> I tried kernel client 4.x and ceph-fuse client with same result.
>
> I'm using erasure for cephfs data pool, cache tier and my storage is
> bluestore and filestore mixed.
>
> How can I help to debug or what should I do to help to find the problem?
>
> With regards
> Jan Pekar
>
>
> On 14.12.2017 15:41, Yan, Zheng wrote:
>>
>> On Thu, Dec 14, 2017 at 8:52 PM, Florent B  wrote:
>>>
>>> On 14/12/2017 03:38, Yan, Zheng wrote:

 On Thu, Dec 14, 2017 at 12:49 AM, Florent B  wrote:
>
>
> Systems are on Debian Jessie : kernel 3.16.0-4-amd64 & libfuse
> 2.9.3-15.
>
> I don't know pattern of corruption, but according to error message in
> Dovecot, it seems to expect data to read but reach EOF.
>
> All seems fine using fuse_disable_pagecache (no more corruption, and
> performance increased : no more MDS slow requests on filelock
> requests).


 I checked ceph-fuse changes since kraken, didn't find any clue. I
 would be helpful if you can try recent version kernel.

 Regards
 Yan, Zheng
>>>
>>>
>>> Problem occurred this morning even with fuse_disable_pagecache=true.
>>>
>>> It seems to be a lock issue between imap & lmtp processes.
>>>
>>> Dovecot uses fcntl as locking method. Is there any change about it in
>>> Luminous ? I switched to flock to see if problem is still there...
>>>
>>
>> I don't remenber there is any change.
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
> --
> 
> Ing. Jan Pekař
> jan.pe...@imatic.cz | +420603811737
> 
> Imatic | Jagellonská 14 | Praha 3 | 130 00
> http://www.imatic.cz
> 
> --
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Corrupted files on CephFS since Luminous upgrade

[Jan Pekař - Imatic]

Hi all,

thank you for reply. I will answer your questions, try to reproduce it 
and if I succeed, start new thread. It can take a while, I'm quiet busy.


My cluster was upgraded from Hammer or Jewel.

Luminous cluster was healthy wen I started my test. It could happen, 
that load temporarily caused health change (osd out and back in). After 
test, my cluster was again healthy.


My first test was from separate client with cephfs kernel client. Second 
test was local cephfs-fuse copying. Both created some file(s) with md5fail.


Before I run tests again, I will upgrade to latest Luminous and let you 
know.


With regards
Jan Pekar

On 28.2.2018 15:14, David C wrote:



On 27 Feb 2018 06:46, "Jan Pekař - Imatic" > wrote:


I think I hit the same issue.
I have corrupted data on cephfs and I don't remember the same issue
before Luminous (i did the same tests before).

It is on my test 1 node cluster with lower memory then recommended
(so server is swapping) but it shouldn't lose data (it never did
before).
So slow requests may appear in the log like Florent B mentioned.

My test is to take some bigger files (few GB) and copy it to cephfs
or from cephfs to cephfs and stress the cluster so data copying
stall for a while. It will resume in few seconds/minutes and
everything looks ok (no error on copying). But copied file may be
corrupted silently.

I checked wiles with MD5SUM and compared some corrupted files in
detail. There were missing some 4MB blocks of data (cephfs object
size) - corrupted file had that block of data filled with zeroes.

My idea is, that there happen something wrong when cluster is under
pressure and client want to save the block. Client gets OK and
continues with another block so data is lost and corrupted block is
filled with zeros.

I tried kernel client 4.x and ceph-fuse client with same result.

I'm using erasure for cephfs data pool, cache tier and my storage is
bluestore and filestore mixed.

How can I help to debug or what should I do to help to find the problem?


Always worrying to see the dreaded C word. I operate a Luminous cluster 
with a pretty varied workload and have yet to see any signs of 
corruption, although of course that doesn't mean its not happening. 
Initial questions:


- What's the history of your cluster? Was this an upgrade or fresh 
Luminous install?

- Was ceph healthy when you ran this test?
-Are you accessing this one node cluster from the node itself or from a 
separate client?


I'd recommend starting a new thread with more details, it sounds like 
it's pretty reproducable for you so maybe crank up your debugging and 
send logs. 
http://docs.ceph.com/docs/luminous/dev/kernel-client-troubleshooting/



With regards
Jan Pekar


On 14.12.2017 15:41, Yan, Zheng wrote:

On Thu, Dec 14, 2017 at 8:52 PM, Florent B > wrote:

On 14/12/2017 03:38, Yan, Zheng wrote:

On Thu, Dec 14, 2017 at 12:49 AM, Florent B
> wrote:


Systems are on Debian Jessie : kernel 3.16.0-4-amd64
& libfuse 2.9.3-15.

I don't know pattern of corruption, but according to
error message in
Dovecot, it seems to expect data to read but reach EOF.

All seems fine using fuse_disable_pagecache (no more
corruption, and
performance increased : no more MDS slow requests on
filelock requests).


I checked ceph-fuse changes since kraken, didn't find
any clue. I
would be helpful if you can try recent version kernel.

Regards
Yan, Zheng


Problem occurred this morning even with
fuse_disable_pagecache=true.

It seems to be a lock issue between imap & lmtp processes.

Dovecot uses fcntl as locking method. Is there any change
about it in
Luminous ? I switched to flock to see if problem is still
there...


I don't remenber there is any change.
___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 


Ing. Jan Pekař
jan.pe...@imatic.cz  | +420603811737


Imatic | Jagellonská 14 | Praha 3 | 130 00
http://www.imatic.cz

--

___
ceph-users mailing list

Re: [ceph-users] Corrupted files on CephFS since Luminous upgrade

[David C]

On 27 Feb 2018 06:46, "Jan Pekař - Imatic"  wrote:

I think I hit the same issue.
I have corrupted data on cephfs and I don't remember the same issue before
Luminous (i did the same tests before).

It is on my test 1 node cluster with lower memory then recommended (so
server is swapping) but it shouldn't lose data (it never did before).
So slow requests may appear in the log like Florent B mentioned.

My test is to take some bigger files (few GB) and copy it to cephfs or from
cephfs to cephfs and stress the cluster so data copying stall for a while.
It will resume in few seconds/minutes and everything looks ok (no error on
copying). But copied file may be corrupted silently.

I checked wiles with MD5SUM and compared some corrupted files in detail.
There were missing some 4MB blocks of data (cephfs object size) - corrupted
file had that block of data filled with zeroes.

My idea is, that there happen something wrong when cluster is under
pressure and client want to save the block. Client gets OK and continues
with another block so data is lost and corrupted block is filled with zeros.

I tried kernel client 4.x and ceph-fuse client with same result.

I'm using erasure for cephfs data pool, cache tier and my storage is
bluestore and filestore mixed.

How can I help to debug or what should I do to help to find the problem?


Always worrying to see the dreaded C word. I operate a Luminous cluster
with a pretty varied workload and have yet to see any signs of corruption,
although of course that doesn't mean its not happening. Initial questions:

- What's the history of your cluster? Was this an upgrade or fresh Luminous
install?
- Was ceph healthy when you ran this test?
-Are you accessing this one node cluster from the node itself or from a
separate client?

I'd recommend starting a new thread with more details, it sounds like it's
pretty reproducable for you so maybe crank up your debugging and send logs.
http://docs.ceph.com/docs/luminous/dev/kernel-client-troubleshooting/


With regards
Jan Pekar


On 14.12.2017 15:41, Yan, Zheng wrote:

> On Thu, Dec 14, 2017 at 8:52 PM, Florent B  wrote:
>
>> On 14/12/2017 03:38, Yan, Zheng wrote:
>>
>>> On Thu, Dec 14, 2017 at 12:49 AM, Florent B  wrote:
>>>

 Systems are on Debian Jessie : kernel 3.16.0-4-amd64 & libfuse 2.9.3-15.

 I don't know pattern of corruption, but according to error message in
 Dovecot, it seems to expect data to read but reach EOF.

 All seems fine using fuse_disable_pagecache (no more corruption, and
 performance increased : no more MDS slow requests on filelock requests).

>>>
>>> I checked ceph-fuse changes since kraken, didn't find any clue. I
>>> would be helpful if you can try recent version kernel.
>>>
>>> Regards
>>> Yan, Zheng
>>>
>>
>> Problem occurred this morning even with fuse_disable_pagecache=true.
>>
>> It seems to be a lock issue between imap & lmtp processes.
>>
>> Dovecot uses fcntl as locking method. Is there any change about it in
>> Luminous ? I switched to flock to see if problem is still there...
>>
>>
> I don't remenber there is any change.
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
-- 

Ing. Jan Pekař
jan.pe...@imatic.cz | +420603811737

Imatic | Jagellonská 14 | Praha 3 | 130 00
http://www.imatic.cz

--

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Kll
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Corrupted files on CephFS since Luminous upgrade

[Jan Pekař - Imatic]

I think I hit the same issue.
I have corrupted data on cephfs and I don't remember the same issue 
before Luminous (i did the same tests before).


It is on my test 1 node cluster with lower memory then recommended (so 
server is swapping) but it shouldn't lose data (it never did before).

So slow requests may appear in the log like Florent B mentioned.

My test is to take some bigger files (few GB) and copy it to cephfs or 
from cephfs to cephfs and stress the cluster so data copying stall for a 
while. It will resume in few seconds/minutes and everything looks ok (no 
error on copying). But copied file may be corrupted silently.


I checked wiles with MD5SUM and compared some corrupted files in detail. 
There were missing some 4MB blocks of data (cephfs object size) - 
corrupted file had that block of data filled with zeroes.


My idea is, that there happen something wrong when cluster is under 
pressure and client want to save the block. Client gets OK and continues 
with another block so data is lost and corrupted block is filled with zeros.


I tried kernel client 4.x and ceph-fuse client with same result.

I'm using erasure for cephfs data pool, cache tier and my storage is 
bluestore and filestore mixed.


How can I help to debug or what should I do to help to find the problem?

With regards
Jan Pekar

On 14.12.2017 15:41, Yan, Zheng wrote:

On Thu, Dec 14, 2017 at 8:52 PM, Florent B  wrote:

On 14/12/2017 03:38, Yan, Zheng wrote:

On Thu, Dec 14, 2017 at 12:49 AM, Florent B  wrote:


Systems are on Debian Jessie : kernel 3.16.0-4-amd64 & libfuse 2.9.3-15.

I don't know pattern of corruption, but according to error message in
Dovecot, it seems to expect data to read but reach EOF.

All seems fine using fuse_disable_pagecache (no more corruption, and
performance increased : no more MDS slow requests on filelock requests).


I checked ceph-fuse changes since kraken, didn't find any clue. I
would be helpful if you can try recent version kernel.

Regards
Yan, Zheng


Problem occurred this morning even with fuse_disable_pagecache=true.

It seems to be a lock issue between imap & lmtp processes.

Dovecot uses fcntl as locking method. Is there any change about it in
Luminous ? I switched to flock to see if problem is still there...



I don't remenber there is any change.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



--

Ing. Jan Pekař
jan.pe...@imatic.cz | +420603811737

Imatic | Jagellonská 14 | Praha 3 | 130 00
http://www.imatic.cz

--
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Corrupted files on CephFS since Luminous upgrade

[Yan, Zheng]

On Thu, Jan 18, 2018 at 6:39 PM, Florent B  wrote:
> I still have file corruption on Ceph-fuse with Luminous (on Debian
> Jessie, default kernel) !
>
> My mounts are using fuse_disable_pagecache=true
>
> And I have a lot of errors like "EOF reading msg header (got 0/30
> bytes)" in my app.

does this happen on kernel client? could you try running ceph-fuse on
recent version kernel.

>
> I didn't have this before Luminous for sure. What debug information can
> I bring to devs to help ?

Can you identify the corrupted file?

> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Corrupted files on CephFS since Luminous upgrade

[Yan, Zheng]

On Thu, Dec 14, 2017 at 8:52 PM, Florent B  wrote:
> On 14/12/2017 03:38, Yan, Zheng wrote:
>> On Thu, Dec 14, 2017 at 12:49 AM, Florent B  wrote:
>>>
>>> Systems are on Debian Jessie : kernel 3.16.0-4-amd64 & libfuse 2.9.3-15.
>>>
>>> I don't know pattern of corruption, but according to error message in
>>> Dovecot, it seems to expect data to read but reach EOF.
>>>
>>> All seems fine using fuse_disable_pagecache (no more corruption, and
>>> performance increased : no more MDS slow requests on filelock requests).
>>
>> I checked ceph-fuse changes since kraken, didn't find any clue. I
>> would be helpful if you can try recent version kernel.
>>
>> Regards
>> Yan, Zheng
>
> Problem occurred this morning even with fuse_disable_pagecache=true.
>
> It seems to be a lock issue between imap & lmtp processes.
>
> Dovecot uses fcntl as locking method. Is there any change about it in
> Luminous ? I switched to flock to see if problem is still there...
>

I don't remenber there is any change.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Corrupted files on CephFS since Luminous upgrade

[Yan, Zheng]

On Thu, Dec 14, 2017 at 12:49 AM, Florent B  wrote:
> On 13/12/2017 17:40, Yan, Zheng wrote:
>> On Wed, Dec 13, 2017 at 11:49 PM, Florent B  wrote:
>>> On 13/12/2017 16:48, Yan, Zheng wrote:
 On Wed, Dec 13, 2017 at 11:23 PM, Florent B  wrote:
> The problem is : only a single client accesses each file !
>
 do you mean the file is only accessed by one client from beginning to end?


> It is not related to a multiple clients accessing a same file at the
> same time, at all.
 The problem can happen even multiples clients accessing a file at
 different time.

>>> Only single client from beginning to end. Each mail account is assigned
>>> to a backend (statically).
>> kernel and libfuse version? do you know pattern of corruption?
>
> Systems are on Debian Jessie : kernel 3.16.0-4-amd64 & libfuse 2.9.3-15.
>
> I don't know pattern of corruption, but according to error message in
> Dovecot, it seems to expect data to read but reach EOF.
>
> All seems fine using fuse_disable_pagecache (no more corruption, and
> performance increased : no more MDS slow requests on filelock requests).


I checked ceph-fuse changes since kraken, didn't find any clue. I
would be helpful if you can try recent version kernel.

Regards
Yan, Zheng
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Corrupted files on CephFS since Luminous upgrade

[Yan, Zheng]

On Wed, Dec 13, 2017 at 11:49 PM, Florent B  wrote:
> On 13/12/2017 16:48, Yan, Zheng wrote:
>> On Wed, Dec 13, 2017 at 11:23 PM, Florent B  wrote:
>>> The problem is : only a single client accesses each file !
>>>
>> do you mean the file is only accessed by one client from beginning to end?
>>
>>
>>> It is not related to a multiple clients accessing a same file at the
>>> same time, at all.
>> The problem can happen even multiples clients accessing a file at
>> different time.
>>
>
> Only single client from beginning to end. Each mail account is assigned
> to a backend (statically).

kernel and libfuse version? do you know pattern of corruption?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Corrupted files on CephFS since Luminous upgrade

[Yan, Zheng]

On Wed, Dec 13, 2017 at 11:23 PM, Florent B  wrote:
> The problem is : only a single client accesses each file !
>

do you mean the file is only accessed by one client from beginning to end?


> It is not related to a multiple clients accessing a same file at the
> same time, at all.

The problem can happen even multiples clients accessing a file at
different time.

Regards
Yan, Zheng
>
> It seems my problem is gone when I set fuse_disable_pagecache to true,
> only on the client accessing this file.
>
> Is it possible a corruption occurs on CephFS ? I never had this problem
> before my Luminous upgrade ! Nothing changed on mail servers (no Dovecot
> upgrade).
>
>
> On 11/12/2017 15:25, Denes Dolhay wrote:
>> Hi,
>>
>>
>> The ceph mds keeps all the capabilities for the files, however the
>> clients modify the the rados data pool objects directly (they do not
>> do the content modification threw the mds).
>>
>> IMHO IF the file (really) gets corrupted because a client write (not
>> some corruption from the mds / osd) then it can only happen if:
>>
>> -The first client does not request write cap (lock) for that object
>> before write
>>
>> -OR the MDS does not store that write cap
>>
>> -OR the MDS does not return the cap to the second client or refuse the
>> write for the second concurrent client
>>
>> -OR the second client does not (request a write cap / check existing
>> caps / does not obey result for write request denial from the mds)
>>
>> -OR any of the clients write incorrect data based on obsolete object
>> cache caused by missing / faulty cache eviction (Is this even possible?)
>>
>> *Please correct me if I am wrong in any of the above!!*
>>
>>
>> If I were in your shoe, first I would test the locking of the cephfs
>> by writing two test scripts:
>>
>> -One would constantly append to a file (like an SMTP server does a
>> mailbox)
>>
>> -The other would modify / add / delete parts o this file (like a imap
>> server does)
>>
>> And wait for corruption to occur
>>
>>
>> One other thing, it would be interesting to see what the corruption
>> really looks like for example partially overwritten lines?
>>
>> It would be interesting to know at what part of the file is the
>> corruption in? beginning? end? %?
>>
>> Aaand if there were a mailbox compact around the corruption
>>
>>
>> Kind regards,
>>
>> Denes.
>>
>>
>>
>> On 12/11/2017 10:33 AM, Florent B wrote:
>>> On 08/12/2017 14:59, Ronny Aasen wrote:
 On 08. des. 2017 14:49, Florent B wrote:
> On 08/12/2017 14:29, Yan, Zheng wrote:
>> On Fri, Dec 8, 2017 at 6:51 PM, Florent B 
>> wrote:
>>> I don't know I didn't touched that setting. Which one is
>>> recommended ?
>>>
>>>
>> If multiple dovecot instances are running at the same time and they
>> all modify the same files. you need to set fuse_disable_pagecache to
>> true.
> Ok, but in my configuration, each mail user is mapped to a single
> server.
> So files are accessed only by a single server at a time.

 how about mail delivery ? if you use dovecot deliver a delivery can
 occur (and rewrite dovecot index/cache) at the same time as a user
 accesses imap and writes to dovecot index/cache.

>>> Ok why not, but I never had problem like this with previous version of
>>> Ceph. I will try fuse_disable_pagecache...
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Corrupted files on CephFS since Luminous upgrade

[Denes Dolhay]

Hi,


The ceph mds keeps all the capabilities for the files, however the 
clients modify the the rados data pool objects directly (they do not do 
the content modification threw the mds).


IMHO IF the file (really) gets corrupted because a client write (not 
some corruption from the mds / osd) then it can only happen if:


-The first client does not request write cap (lock) for that object 
before write


-OR the MDS does not store that write cap

-OR the MDS does not return the cap to the second client or refuse the 
write for the second concurrent client


-OR the second client does not (request a write cap / check existing 
caps / does not obey result for write request denial from the mds)


-OR any of the clients write incorrect data based on obsolete object 
cache caused by missing / faulty cache eviction (Is this even possible?)


*Please correct me if I am wrong in any of the above!!*


If I were in your shoe, first I would test the locking of the cephfs by 
writing two test scripts:


-One would constantly append to a file (like an SMTP server does a mailbox)

-The other would modify / add / delete parts o this file (like a imap 
server does)


And wait for corruption to occur


One other thing, it would be interesting to see what the corruption 
really looks like for example partially overwritten lines?


It would be interesting to know at what part of the file is the 
corruption in? beginning? end? %?


Aaand if there were a mailbox compact around the corruption


Kind regards,

Denes.



On 12/11/2017 10:33 AM, Florent B wrote:

On 08/12/2017 14:59, Ronny Aasen wrote:

On 08. des. 2017 14:49, Florent B wrote:

On 08/12/2017 14:29, Yan, Zheng wrote:

On Fri, Dec 8, 2017 at 6:51 PM, Florent B  wrote:

I don't know I didn't touched that setting. Which one is recommended ?



If multiple dovecot instances are running at the same time and they
all modify the same files. you need to set fuse_disable_pagecache to
true.

Ok, but in my configuration, each mail user is mapped to a single
server.
So files are accessed only by a single server at a time.


how about mail delivery ? if you use dovecot deliver a delivery can
occur (and rewrite dovecot index/cache) at the same time as a user
accesses imap and writes to dovecot index/cache.


Ok why not, but I never had problem like this with previous version of
Ceph. I will try fuse_disable_pagecache...
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Corrupted files on CephFS since Luminous upgrade

[Yan, Zheng]

On Fri, Dec 8, 2017 at 10:04 PM, Florent B  wrote:
> When I look in MDS slow requests I have a few like this :
>
> {
> "description": "client_request(client.460346000:5211
> setfilelockrule 1, type 2, owner 9688352835732396778, pid 660, start 0,
> length 0, wait 1 #0x100017da2aa 2017-12-08 15:00:21.823976
> caller_uid=5000, caller_gid=5000{5000,})",
> "initiated_at": "2017-12-08 15:00:21.834698",
> "age": 72.168096,
> "duration": 72.168212,
> "type_data": {
> "flag_point": "acquired locks",
> "reqid": "client.460346000:5211",
> "op_type": "client_request",
> "client_info": {
> "client": "client.460346000",
> "tid": 5211
> },
> "events": [
> {
> "time": "2017-12-08 15:00:21.834698",
> "event": "initiated"
> },
> {
> "time": "2017-12-08 15:00:21.834826",
> "event": "acquired locks"
> },
> {
> "time": "2017-12-08 15:00:28.299564",
> "event": "acquired locks"
> },
> {
> "time": "2017-12-08 15:00:50.720873",
> "event": "acquired locks"
> },
> {
> "time": "2017-12-08 15:01:05.884944",
> "event": "acquired locks"
> },
> {
> "time": "2017-12-08 15:01:17.043605",
> "event": "acquired locks"
> }
> ]
> }
>
> Always for the same client (my mail server).

This is file lock request. It blocks until lock holder releases the
lock (maybe we should not print waring for requests of this type).

you can dump mds cache and check if the corrupted file has caps from
multiple clients.


> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Corrupted files on CephFS since Luminous upgrade

[Ronny Aasen]

On 08. des. 2017 14:49, Florent B wrote:

On 08/12/2017 14:29, Yan, Zheng wrote:

On Fri, Dec 8, 2017 at 6:51 PM, Florent B  wrote:

I don't know I didn't touched that setting. Which one is recommended ?



If multiple dovecot instances are running at the same time and they
all modify the same files. you need to set fuse_disable_pagecache to
true.


Ok, but in my configuration, each mail user is mapped to a single server.
So files are accessed only by a single server at a time.



how about mail delivery ? if you use dovecot deliver a delivery can 
occur (and rewrite dovecot index/cache) at the same time as a user 
accesses imap and writes to dovecot index/cache.




kind regards
Ronny Aasen

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Corrupted files on CephFS since Luminous upgrade

[Yan, Zheng]

On Fri, Dec 8, 2017 at 6:51 PM, Florent B <flor...@coppint.com> wrote:
> I don't know I didn't touched that setting. Which one is recommended ?
>
>

If multiple dovecot instances are running at the same time and they
all modify the same files. you need to set fuse_disable_pagecache to
true.

> On 08/12/2017 11:49, Alexandre DERUMIER wrote:
>> do you have disabled fuse pagecache on your clients ceph.conf ?
>>
>>
>> [client]
>> fuse_disable_pagecache = true
>>
>> - Mail original -
>> De: "Florent Bautista" <flor...@coppint.com>
>> À: "ceph-users" <ceph-users@lists.ceph.com>
>> Envoyé: Vendredi 8 Décembre 2017 10:54:59
>> Objet: Re: [ceph-users] Corrupted files on CephFS since Luminous upgrade
>>
>> On 08/12/2017 10:44, Wido den Hollander wrote:
>>>
>>> On 12/08/2017 10:27 AM, Florent B wrote:
>>>> Hi everyone,
>>>>
>>>> A few days ago I upgraded a cluster from Kraken to Luminous.
>>>>
>>>> I have a few mail servers on it, running Ceph-Fuse & Dovecot.
>>>>
>>>> And since the day of upgrade, Dovecot is reporting some corrupted files
>>>> on a large account :
>>>>
>>>> doveadm(myu...@mydomain.com): Error: Corrupted dbox file
>>>> /mnt/maildata1/mydomain.com/myuser//mdbox/storage/m.5808 (around
>>>> offset=79178): purging found mismatched offsets (79148 vs 72948,
>>>> 13/1313)
>>>> doveadm(myu...@mydomain.com): Warning: fscking index file
>>>> /mnt/maildata1/mydomain.com/myuser//mdbox/storage/dovecot.map.index
>>>> doveadm(myu...@mydomain.com): Warning: mdbox
>>>> /mnt/maildata1/mydomain.com/myuser//mdbox/storage: rebuilding indexes
>>>> doveadm(myu...@mydomain.com): Warning: Transaction log file
>>>> /mnt/maildata1/mydomain.com/myuser//mdbox/storage/dovecot.map.index.log
>>>> was locked for 1249 seconds (mdbox storage rebuild)
>>>> doveadm(myu...@mydomain.com): Error: Purging namespace '' failed:
>>>> Corrupted dbox file
>>>> /mnt/maildata1/mydomain.com/myuser//mdbox/storage/m.5808 (around
>>>> offset=79178): purging found mismatched offsets (79148 vs 72948,
>>>> 13/1313)
>>>>
>>>> Even if Dovecot fixes this problem, every day new files are corrupted.
>>>>
>>>> I never had this problem before ! And Ceph status is reporting some "MDS
>>>> slow requests" !
>>>>
>>>> Do you have an idea ?
>>>>
>>> Not really, but could you share a bit more information:
>>>
>>> - Which version if Luminous?
>>> - Running with BlueStore or FileStore?
>>> - Replication?
>>> - Cache tiering?
>>> - Which kernel version do the clients use?
>>>
>>> Wido
>>>
>> Luminous 12.2.1 upgraded to 12.2.2 yesterday, and always the same
>> problem today.
>>
>> FileStore only (xfs).
>>
>> Replication is 3 copies for these mail files.
>>
>> No Cache Tiering.
>>
>> Kernel on clients is default Debian Jessie (3.16.43-2+deb8u5) but I'm
>> using ceph-fuse, not kernel client.
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Corrupted files on CephFS since Luminous upgrade

[Alexandre DERUMIER]

do you have disabled fuse pagecache on your clients ceph.conf ?


[client]
fuse_disable_pagecache = true

- Mail original -
De: "Florent Bautista" <flor...@coppint.com>
À: "ceph-users" <ceph-users@lists.ceph.com>
Envoyé: Vendredi 8 Décembre 2017 10:54:59
Objet: Re: [ceph-users] Corrupted files on CephFS since Luminous upgrade

On 08/12/2017 10:44, Wido den Hollander wrote: 
> 
> 
> On 12/08/2017 10:27 AM, Florent B wrote: 
>> Hi everyone, 
>> 
>> A few days ago I upgraded a cluster from Kraken to Luminous. 
>> 
>> I have a few mail servers on it, running Ceph-Fuse & Dovecot. 
>> 
>> And since the day of upgrade, Dovecot is reporting some corrupted files 
>> on a large account : 
>> 
>> doveadm(myu...@mydomain.com): Error: Corrupted dbox file 
>> /mnt/maildata1/mydomain.com/myuser//mdbox/storage/m.5808 (around 
>> offset=79178): purging found mismatched offsets (79148 vs 72948, 
>> 13/1313) 
>> doveadm(myu...@mydomain.com): Warning: fscking index file 
>> /mnt/maildata1/mydomain.com/myuser//mdbox/storage/dovecot.map.index 
>> doveadm(myu...@mydomain.com): Warning: mdbox 
>> /mnt/maildata1/mydomain.com/myuser//mdbox/storage: rebuilding indexes 
>> doveadm(myu...@mydomain.com): Warning: Transaction log file 
>> /mnt/maildata1/mydomain.com/myuser//mdbox/storage/dovecot.map.index.log 
>> was locked for 1249 seconds (mdbox storage rebuild) 
>> doveadm(myu...@mydomain.com): Error: Purging namespace '' failed: 
>> Corrupted dbox file 
>> /mnt/maildata1/mydomain.com/myuser//mdbox/storage/m.5808 (around 
>> offset=79178): purging found mismatched offsets (79148 vs 72948, 
>> 13/1313) 
>> 
>> Even if Dovecot fixes this problem, every day new files are corrupted. 
>> 
>> I never had this problem before ! And Ceph status is reporting some "MDS 
>> slow requests" ! 
>> 
>> Do you have an idea ? 
>> 
> 
> Not really, but could you share a bit more information: 
> 
> - Which version if Luminous? 
> - Running with BlueStore or FileStore? 
> - Replication? 
> - Cache tiering? 
> - Which kernel version do the clients use? 
> 
> Wido 
> 

Luminous 12.2.1 upgraded to 12.2.2 yesterday, and always the same 
problem today. 

FileStore only (xfs). 

Replication is 3 copies for these mail files. 

No Cache Tiering. 

Kernel on clients is default Debian Jessie (3.16.43-2+deb8u5) but I'm 
using ceph-fuse, not kernel client. 
___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Corrupted files on CephFS since Luminous upgrade

[Wido den Hollander]

On 12/08/2017 10:27 AM, Florent B wrote:

Hi everyone,

A few days ago I upgraded a cluster from Kraken to Luminous.

I have a few mail servers on it, running Ceph-Fuse & Dovecot.

And since the day of upgrade, Dovecot is reporting some corrupted files
on a large account :

doveadm(myu...@mydomain.com): Error: Corrupted dbox file
/mnt/maildata1/mydomain.com/myuser//mdbox/storage/m.5808 (around
offset=79178): purging found mismatched offsets (79148 vs 72948, 13/1313)
doveadm(myu...@mydomain.com): Warning: fscking index file
/mnt/maildata1/mydomain.com/myuser//mdbox/storage/dovecot.map.index
doveadm(myu...@mydomain.com): Warning: mdbox
/mnt/maildata1/mydomain.com/myuser//mdbox/storage: rebuilding indexes
doveadm(myu...@mydomain.com): Warning: Transaction log file
/mnt/maildata1/mydomain.com/myuser//mdbox/storage/dovecot.map.index.log
was locked for 1249 seconds (mdbox storage rebuild)
doveadm(myu...@mydomain.com): Error: Purging namespace '' failed:
Corrupted dbox file
/mnt/maildata1/mydomain.com/myuser//mdbox/storage/m.5808 (around
offset=79178): purging found mismatched offsets (79148 vs 72948, 13/1313)

Even if Dovecot fixes this problem, every day new files are corrupted.

I never had this problem before ! And Ceph status is reporting some "MDS
slow requests" !

Do you have an idea ?



Not really, but could you share a bit more information:

- Which version if Luminous?
- Running with BlueStore or FileStore?
- Replication?
- Cache tiering?
- Which kernel version do the clients use?

Wido



Thank you

Florent

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com