Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[Stefan Kooman]

Hi,

Quoting Stefan Kooman (ste...@bit.nl):
> Hi,
> 
> We see the following in the logs after we start a scrub for some osds:
> 
> ceph-osd.2.log:2017-12-14 06:50:47.180344 7f0f47db2700  0 
> log_channel(cluster) log [DBG] : 1.2d8 scrub starts
> ceph-osd.2.log:2017-12-14 06:50:47.180915 7f0f47db2700 -1 osd.2 pg_epoch: 
> 11897 pg[1.2d8( v 11890'165209 (3221'163647,11890'165209] 
> local-lis/les=11733/11734 n=67 ec=132/132 lis/c 11733/11733 les/c/f 
> 11734/11734/0 11733/11733/11733) [2,45,31] r=0 lpr=11733 crt=11890'165209 
> lcod 11890'165208 mlcod 11890'165208 active+clean+scrubbing] _scan_snaps no 
> head for 1:1b518155:::rbd_data.620652ae8944a.0126:29 (have MIN)
> ceph-osd.2.log:2017-12-14 06:50:47.180929 7f0f47db2700 -1 osd.2 pg_epoch: 
> 11897 pg[1.2d8( v 11890'165209 (3221'163647,11890'165209] 
> local-lis/les=11733/11734 n=67 ec=132/132 lis/c 11733/11733 les/c/f 
> 11734/11734/0 11733/11733/11733) [2,45,31] r=0 lpr=11733 crt=11890'165209 
> lcod 11890'165208 mlcod 11890'165208 active+clean+scrubbing] _scan_snaps no 
> head for 1:1b518155:::rbd_data.620652ae8944a.0126:14 (have MIN)
> ceph-osd.2.log:2017-12-14 06:50:47.180941 7f0f47db2700 -1 osd.2 pg_epoch: 
> 11897 pg[1.2d8( v 11890'165209 (3221'163647,11890'165209] 
> local-lis/les=11733/11734 n=67 ec=132/132 lis/c 11733/11733 les/c/f 
> 11734/11734/0 11733/11733/11733) [2,45,31] r=0 lpr=11733 crt=11890'165209 
> lcod 11890'165208 mlcod 11890'165208 active+clean+scrubbing] _scan_snaps no 
> head for 1:1b518155:::rbd_data.620652ae8944a.0126:a (have MIN)
> ceph-osd.2.log:2017-12-14 06:50:47.214198 7f0f43daa700  0 
> log_channel(cluster) log [DBG] : 1.2d8 scrub ok
> 
> So finally it logs "scrub ok", but what does " _scan_snaps no head for ..." 
> mean?
> Does this indicate a problem?

Still seeing this issue on a freshly installed luminous cluster. I
*think* it either has to do with "cloned" RBDs that get snapshots by
themselves or RBDs that are cloned from a snapshot.

Any dev that wants to debug this behaviour if I'm able to reliably
reproduce this?

Gr. Stefan

-- 
| BIT BV  http://www.bit.nl/Kamer van Koophandel 09090351
| GPG: 0xD14839C6   +31 318 648 688 / i...@bit.nl
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[Marc Roos]

 
Oh that is very good to hear. So how should I be cleaning this up? I 
read some post of Sage that scrubbing is not taking care of this. 
Should I be dumping the logs with objects like 
17:e80576a8:::rbd_data.2cc7df2ae8944a.09f8:27  and try to 
delete these manually?




-Original Message-
From: Paul Emmerich [mailto:paul.emmer...@croit.io] 
Sent: donderdag 12 april 2018 11:04
To: Marc Roos
Cc: ceph-users
Subject: Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for 
$object?

Usually the problem is not that you are missing snapshot data, but that 
you got too many snapshots, so your snapshots are probably fine. You're 
just wasting space.



Paul


2018-04-10 16:07 GMT+02:00 Marc Roos <m.r...@f1-outsourcing.eu>:


 
Hi Paul,

This is a small test cluster, and the rbd pool is replicated. I am 
hardly using any clients on the cluster. Furthermore I have been 
the 
only one creating the snapshots and I know for sure that I was not 
trying to delete them. If so I have been doing this on one of the 
ceph 
nodes.

I have these issues on images with
create_timestamp: Tue Jul 18 20:51:40 2017
create_timestamp: Fri Sep  1 13:55:25 2017
create_timestamp: Fri Sep  1 13:59:10 2017
create_timestamp: Wed Jan  3 16:38:57 2018

Updates have been done in February, so theoretically I should not 
be 
seeing these than any more?
Feb 21 15:13:35 Updated: 2:ceph-osd-12.2.3-0.el7.x86_64
Feb 28 13:33:27 Updated: 2:ceph-osd-12.2.4-0.el7.x86_64

How can I determine what snapshot is bad of this image? 
Should this snapshot be considered lost?
And is deleting this snapshot the only way to fix this? 


-Original Message-
From: Paul Emmerich [mailto:paul.emmer...@croit.io] 
Sent: dinsdag 10 april 2018 20:14
To: Marc Roos
Cc: ceph-users
Subject: Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for 
    $object?


Hi,


you'll usually see this if there are "orphaned" snapshot objects. 
One 
common cause for this are
pre-12.2.2 clients trying to delete RBD snapshots with a data pool 
(i.e., erasure coded pools) They send the snapshot requests to the 
wrong 
pool and you end up with lots of problems.



Paul


2018-04-09 16:55 GMT+02:00 Marc Roos <m.r...@f1-outsourcing.eu>:



I have this on a rbd pool with images/snapshots that have 
been 
created
in Luminous

> Hi Stefan, Mehmet,
>
> Are these clusters that were upgraded from prior 
versions, or 
fresh
> luminous installs?
>
>
> This message indicates that there is a stray clone object 
with no
> associated head or snapdir object.  That normally should 
never
> happen--it's presumably the result of a (hopefully old) 
bug.  The
scrub
> process doesn't even clean them up, which maybe says 
something 
about
how
> common it is/was...
>
> sage

>




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>  
<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> > 





-- 

--
Paul Emmerich

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90








-- 

--
Paul Emmerich

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[Paul Emmerich]

Usually the problem is not that you are missing snapshot data, but that you
got
too many snapshots, so your snapshots are probably fine. You're just wasting
space.


Paul

2018-04-10 16:07 GMT+02:00 Marc Roos <m.r...@f1-outsourcing.eu>:

>
> Hi Paul,
>
> This is a small test cluster, and the rbd pool is replicated. I am
> hardly using any clients on the cluster. Furthermore I have been the
> only one creating the snapshots and I know for sure that I was not
> trying to delete them. If so I have been doing this on one of the ceph
> nodes.
>
> I have these issues on images with
> create_timestamp: Tue Jul 18 20:51:40 2017
> create_timestamp: Fri Sep  1 13:55:25 2017
> create_timestamp: Fri Sep  1 13:59:10 2017
> create_timestamp: Wed Jan  3 16:38:57 2018
>
> Updates have been done in February, so theoretically I should not be
> seeing these than any more?
> Feb 21 15:13:35 Updated: 2:ceph-osd-12.2.3-0.el7.x86_64
> Feb 28 13:33:27 Updated: 2:ceph-osd-12.2.4-0.el7.x86_64
>
> How can I determine what snapshot is bad of this image?
> Should this snapshot be considered lost?
> And is deleting this snapshot the only way to fix this?
>
>
> -Original Message-
> From: Paul Emmerich [mailto:paul.emmer...@croit.io]
> Sent: dinsdag 10 april 2018 20:14
> To: Marc Roos
> Cc: ceph-users
> Subject: Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for
> $object?
>
> Hi,
>
>
> you'll usually see this if there are "orphaned" snapshot objects. One
> common cause for this are
> pre-12.2.2 clients trying to delete RBD snapshots with a data pool
> (i.e., erasure coded pools) They send the snapshot requests to the wrong
> pool and you end up with lots of problems.
>
>
>
> Paul
>
>
> 2018-04-09 16:55 GMT+02:00 Marc Roos <m.r...@f1-outsourcing.eu>:
>
>
>
> I have this on a rbd pool with images/snapshots that have been
> created
> in Luminous
>
> > Hi Stefan, Mehmet,
> >
> > Are these clusters that were upgraded from prior versions, or
> fresh
> > luminous installs?
> >
> >
> > This message indicates that there is a stray clone object with no
> > associated head or snapdir object.  That normally should never
> > happen--it's presumably the result of a (hopefully old) bug.  The
> scrub
> > process doesn't even clean them up, which maybe says something
> about
> how
> > common it is/was...
> >
> > sage
>
> >
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
>
>
>
>
>
> --
>
> --
> Paul Emmerich
>
> croit GmbH
> Freseniusstr. 31h
> 81247 München
> www.croit.io
> Tel: +49 89 1896585 90
>
>
>
>


-- 
-- 
Paul Emmerich

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[Marc Roos]

 
Hi Paul,

This is a small test cluster, and the rbd pool is replicated. I am 
hardly using any clients on the cluster. Furthermore I have been the 
only one creating the snapshots and I know for sure that I was not 
trying to delete them. If so I have been doing this on one of the ceph 
nodes.

I have these issues on images with
create_timestamp: Tue Jul 18 20:51:40 2017
create_timestamp: Fri Sep  1 13:55:25 2017
create_timestamp: Fri Sep  1 13:59:10 2017
create_timestamp: Wed Jan  3 16:38:57 2018

Updates have been done in February, so theoretically I should not be 
seeing these than any more?
Feb 21 15:13:35 Updated: 2:ceph-osd-12.2.3-0.el7.x86_64
Feb 28 13:33:27 Updated: 2:ceph-osd-12.2.4-0.el7.x86_64

How can I determine what snapshot is bad of this image? 
Should this snapshot be considered lost?
And is deleting this snapshot the only way to fix this? 


-Original Message-
From: Paul Emmerich [mailto:paul.emmer...@croit.io] 
Sent: dinsdag 10 april 2018 20:14
To: Marc Roos
Cc: ceph-users
Subject: Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for 
$object?

Hi,


you'll usually see this if there are "orphaned" snapshot objects. One 
common cause for this are
pre-12.2.2 clients trying to delete RBD snapshots with a data pool 
(i.e., erasure coded pools) They send the snapshot requests to the wrong 
pool and you end up with lots of problems.



Paul


2018-04-09 16:55 GMT+02:00 Marc Roos <m.r...@f1-outsourcing.eu>:



I have this on a rbd pool with images/snapshots that have been 
created
in Luminous

> Hi Stefan, Mehmet,
>
> Are these clusters that were upgraded from prior versions, or 
fresh
> luminous installs?
>
>
> This message indicates that there is a stray clone object with no
> associated head or snapdir object.  That normally should never
> happen--it's presumably the result of a (hopefully old) bug.  The
scrub
> process doesn't even clean them up, which maybe says something 
about
how
> common it is/was...
>
> sage

>




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> 





-- 

--
Paul Emmerich

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[Paul Emmerich]

Hi,

you'll usually see this if there are "orphaned" snapshot objects. One
common cause for this are
pre-12.2.2 clients trying to delete RBD snapshots with a data pool (i.e.,
erasure coded pools)
They send the snapshot requests to the wrong pool and you end up with lots
of problems.


Paul

2018-04-09 16:55 GMT+02:00 Marc Roos :

>
> I have this on a rbd pool with images/snapshots that have been created
> in Luminous
>
> > Hi Stefan, Mehmet,
> >
> > Are these clusters that were upgraded from prior versions, or fresh
> > luminous installs?
> >
> >
> > This message indicates that there is a stray clone object with no
> > associated head or snapdir object.  That normally should never
> > happen--it's presumably the result of a (hopefully old) bug.  The
> scrub
> > process doesn't even clean them up, which maybe says something about
> how
> > common it is/was...
> >
> > sage
> >
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
-- 
Paul Emmerich

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[Marc Roos]

I have this on a rbd pool with images/snapshots that have been created 
in Luminous

> Hi Stefan, Mehmet,
> 
> Are these clusters that were upgraded from prior versions, or fresh 
> luminous installs?
> 
> 
> This message indicates that there is a stray clone object with no 
> associated head or snapdir object.  That normally should never 
> happen--it's presumably the result of a (hopefully old) bug.  The 
scrub 
> process doesn't even clean them up, which maybe says something about 
how 
> common it is/was...
> 
> sage
> 
 



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[Marc Roos]

 
I have found one image, how do I know what snapshot version to delete? I 
have multiple




-Original Message-
From: c...@elchaka.de [mailto:c...@elchaka.de] 
Sent: zondag 8 april 2018 13:30
To: ceph-users
Subject: Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for 
$object?



Am 8. April 2018 05:44:11 MESZ schrieb Marc Roos 
<m.r...@f1-outsourcing.eu>:
>
>Hi Mehmet,
> 
>The data is already lost in these snapshots?

I cannot say that. Cause i did Not need the Snapshots. But you can try 
to Clone the vm in the state of the Snapshot ( i am using proxmox).  
  
> And how did you identify
>the snapshot? It looks like I have these only in the rbd pool. 

You have to use "rbd info" to identify which Image is Related to this. 
Search for " 239f5274b0dc51"

- Mehmet 
>
>
>
>
>-Original Message-
>From: c...@elchaka.de [mailto:c...@elchaka.de]
>Sent: zondag 8 april 2018 10:44
>To: ceph-users@lists.ceph.com
>Subject: Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for 
>$object?
>
>Hi Marc,
>
>Am 7. April 2018 18:32:40 MESZ schrieb Marc Roos
><m.r...@f1-outsourcing.eu>:
>>
>>How do you resolve these issues?
>>
>
>In my Case i could get rid of this by deleting the existing Snapshots.
>
>- Mehmet   
>>
>>Apr  7 22:39:21 c03 ceph-osd: 2018-04-07 22:39:21.928484 7f0826524700
>>-1
>>osd.13 pg_epoch: 19008 pg[17.13( v 19008'6019891 
>>(19008'6018375,19008'6019891] local-lis/les=18980/18981 n=3825
>>ec=3636/3636 lis/c 18980/18980 les/c/f 18981/18982/0
>18980/18980/18903)
>>
>>[4,13,0] r=1 lpr=18980 luod=0'0 crt=19008'6019891 lcod 19008'6019890 
>>active] _scan_snaps no head for
>>17:cbf61056:::rbd_data.239f5274b0dc51.0ff2:15 (have MIN) 
>>___
>>ceph-users mailing list
>>ceph-users@lists.ceph.com
>>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>___
>ceph-users mailing list
>ceph-users@lists.ceph.com
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[ceph]

Am 8. April 2018 05:44:11 MESZ schrieb Marc Roos <m.r...@f1-outsourcing.eu>:
>
>Hi Mehmet,
> 
>The data is already lost in these snapshots?

I cannot say that. Cause i did Not need the Snapshots. But you can try to Clone 
the vm in the state of the Snapshot ( i am using proxmox).  
  
> And how did you identify 
>the snapshot? It looks like I have these only in the rbd pool. 

You have to use "rbd info" to identify which Image is Related to this. Search 
for " 239f5274b0dc51"

- Mehmet 
>
>
>
>
>-Original Message-
>From: c...@elchaka.de [mailto:c...@elchaka.de] 
>Sent: zondag 8 april 2018 10:44
>To: ceph-users@lists.ceph.com
>Subject: Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for 
>$object?
>
>Hi Marc,
>
>Am 7. April 2018 18:32:40 MESZ schrieb Marc Roos 
><m.r...@f1-outsourcing.eu>:
>>
>>How do you resolve these issues?
>>
>
>In my Case i could get rid of this by deleting the existing Snapshots.
>
>- Mehmet   
>>
>>Apr  7 22:39:21 c03 ceph-osd: 2018-04-07 22:39:21.928484 7f0826524700
>>-1
>>osd.13 pg_epoch: 19008 pg[17.13( v 19008'6019891 
>>(19008'6018375,19008'6019891] local-lis/les=18980/18981 n=3825
>>ec=3636/3636 lis/c 18980/18980 les/c/f 18981/18982/0
>18980/18980/18903)
>>
>>[4,13,0] r=1 lpr=18980 luod=0'0 crt=19008'6019891 lcod 19008'6019890 
>>active] _scan_snaps no head for
>>17:cbf61056:::rbd_data.239f5274b0dc51.0ff2:15 (have MIN) 
>>___
>>ceph-users mailing list
>>ceph-users@lists.ceph.com
>>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>___
>ceph-users mailing list
>ceph-users@lists.ceph.com
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[Marc Roos]

Hi Mehmet,
 
The data is already lost in these snapshots? And how did you identify 
the snapshot? It looks like I have these only in the rbd pool. 




-Original Message-
From: c...@elchaka.de [mailto:c...@elchaka.de] 
Sent: zondag 8 april 2018 10:44
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for 
$object?

Hi Marc,

Am 7. April 2018 18:32:40 MESZ schrieb Marc Roos 
<m.r...@f1-outsourcing.eu>:
>
>How do you resolve these issues?
>

In my Case i could get rid of this by deleting the existing Snapshots.

- Mehmet   
>
>Apr  7 22:39:21 c03 ceph-osd: 2018-04-07 22:39:21.928484 7f0826524700
>-1
>osd.13 pg_epoch: 19008 pg[17.13( v 19008'6019891 
>(19008'6018375,19008'6019891] local-lis/les=18980/18981 n=3825
>ec=3636/3636 lis/c 18980/18980 les/c/f 18981/18982/0 18980/18980/18903)
>
>[4,13,0] r=1 lpr=18980 luod=0'0 crt=19008'6019891 lcod 19008'6019890 
>active] _scan_snaps no head for
>17:cbf61056:::rbd_data.239f5274b0dc51.0ff2:15 (have MIN) 
>___
>ceph-users mailing list
>ceph-users@lists.ceph.com
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[ceph]

Hi Marc,

Am 7. April 2018 18:32:40 MESZ schrieb Marc Roos :
>
>How do you resolve these issues?
>

In my Case i could get rid of this by deleting the existing Snapshots.

- Mehmet   
>
>Apr  7 22:39:21 c03 ceph-osd: 2018-04-07 22:39:21.928484 7f0826524700
>-1 
>osd.13 pg_epoch: 19008 pg[17.13( v 19008'6019891 
>(19008'6018375,19008'6019891] local-lis/les=18980/18981 n=3825 
>ec=3636/3636 lis/c 18980/18980 les/c/f 18981/18982/0 18980/18980/18903)
>
>[4,13,0] r=1 lpr=18980 luod=0'0 crt=19008'6019891 lcod 19008'6019890 
>active] _scan_snaps no head for 
>17:cbf61056:::rbd_data.239f5274b0dc51.0ff2:15 (have MIN)
>___
>ceph-users mailing list
>ceph-users@lists.ceph.com
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[Marc Roos]

How do you resolve these issues?


Apr  7 22:39:21 c03 ceph-osd: 2018-04-07 22:39:21.928484 7f0826524700 -1 
osd.13 pg_epoch: 19008 pg[17.13( v 19008'6019891 
(19008'6018375,19008'6019891] local-lis/les=18980/18981 n=3825 
ec=3636/3636 lis/c 18980/18980 les/c/f 18981/18982/0 18980/18980/18903) 
[4,13,0] r=1 lpr=18980 luod=0'0 crt=19008'6019891 lcod 19008'6019890 
active] _scan_snaps no head for 
17:cbf61056:::rbd_data.239f5274b0dc51.0ff2:15 (have MIN)
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[Mehmet]

Sage Wrote( Tue, 2 Jan 2018 17:57:32 + (UTC)):

Hi Stefan, Mehmet,



Hi Sage,
Sorry for the *extremly late* response!


Are these clusters that were upgraded from prior versions, or fresh
luminous installs?


My Cluster was initialy installed with jewel (10.2.1) have seen some 
minor updates and is finaly upgraded from Jewel (10.2.10) to Luminous 
(12.2.1)


Actualy is installed:

- ceph version 12.2.2 (cf0baba3b47f9427c6c97e2144b094b7e5ba) 
luminous (stable)


I had a look in my logfiled and have still the log entries like:

... .. .
2018-02-23 11:23:34.247878 7feaa2a2d700 -1 osd.59 pg_epoch: 36269 
pg[0.346( v 36269'30160204 (36269'30158634,36269'30160204] 
local-lis/les=36253/36254 n=12956 ec=141/141 lis/c 36253/36253 les/c/f 
36254/36264/0 36253/36253/36190) [4,59,23] r=1 lpr=36253 luod=0'0 
crt=36269'30160204 lcod 36269'30160203 active] _scan_snaps no head for 
0:62e347cd:::rbd_data.63efee238e1f29.038c:48 (have MIN)

... .. .

need further information?
- Mehmet



This message indicates that there is a stray clone object with no
associated head or snapdir object.  That normally should never
happen--it's presumably the result of a (hopefully old) bug.  The scrub
process doesn't even clean them up, which maybe says something about 
how

common it is/was...

sage


On Sun, 24 Dec 2017, ceph@xx wrote:

> Hi Stefan,
>
> Am 14. Dezember 2017 09:48:36 MEZ schrieb Stefan Kooman :
> >Hi,
> >
> >We see the following in the logs after we start a scrub for some osds:
> >
> >ceph-osd.2.log:2017-12-14 06:50:47.180344 7f0f47db2700  0
> >log_channel(cluster) log [DBG] : 1.2d8 scrub starts
> >ceph-osd.2.log:2017-12-14 06:50:47.180915 7f0f47db2700 -1 osd.2
> >pg_epoch: 11897 pg[1.2d8( v 11890'165209 (3221'163647,11890'165209]
> >local-lis/les=11733/11734 n=67 ec=132/132 lis/c 11733/11733 les/c/f
> > >11734/11734/0 11733/11733/11733) [2,45,31] r=0 lpr=11733
> >crt=11890'165209 lcod 11890'165208 mlcod 11890'165208
> >active+clean+scrubbing] _scan_snaps no head for
> >1:1b518155:::rbd_data.620652ae8944a.0126:29 (have MIN)
> >ceph-osd.2.log:2017-12-14 06:50:47.180929 7f0f47db2700 -1 osd.2
> >pg_epoch: 11897 pg[1.2d8( v 11890'165209 (3221'163647,11890'165209]
> >local-lis/les=11733/11734 n=67 ec=132/132 lis/c 11733/11733 les/c/f
> >11734/11734/0 11733/11733/11733) [2,45,31] r=0 lpr=11733
> >crt=11890'165209 lcod 11890'165208 mlcod 11890'165208
> >active+clean+scrubbing] _scan_snaps no head for
> >1:1b518155:::rbd_data.620652ae8944a.0126:14 (have MIN)
> >ceph-osd.2.log:2017-12-14 06:50:47.180941 7f0f47db2700 -1 osd.2
> >pg_epoch: 11897 pg[1.2d8( v 11890'165209 (3221'163647,11890'165209]
> >local-lis/les=11733/11734 n=67 ec=132/132 lis/c 11733/11733 les/c/f
> >11734/11734/0 11733/11733/11733) [2,45,31] r=0 lpr=11733
> >crt=11890'165209 lcod 11890'165208 mlcod 11890'165208
> >active+clean+scrubbing] _scan_snaps no head for
> >1:1b518155:::rbd_data.620652ae8944a.0126:a (have MIN)
> >ceph-osd.2.log:2017-12-14 06:50:47.214198 7f0f43daa700  0
> >log_channel(cluster) log [DBG] : 1.2d8 scrub ok
> >
> >So finally it logs "scrub ok", but what does " _scan_snaps no head for
> >..." mean?
>
> I also see this lines in our Logfiles and am wonder  what this means.
>
> >Does this indicate a problem?
>
> I do not guess so because we actually have not  any issues.
>
> >
> >Ceph 12.2.2 with bluestore on lvm
>
> We using 12.2.2 with filestore on xfs.
>
> - Mehmet
> >
> >Gr. Stefan
> ___
> ceph-users mailing list

v> ceph-users@xx

> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>


___
ceph-users mailing list
ceph-users@xx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[Konstantin Shalygin]

On 01/04/2018 11:53 PM, Stefan Kooman wrote:

OpenNebula 5.4.3 (issuing rbd commands to ceph cluster).


Yes! And what librbd is installed on "commands issuer"?



k
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[Stefan Kooman]

Quoting Konstantin Shalygin (k0...@k0ste.ru):
> On 01/04/2018 11:38 PM, Stefan Kooman wrote:
> >Only luminous clients. Mostly rbd (qemu-kvm) images.
> 
> Who is managed your images? May be OpenStack Cinder?

OpenNebula 5.4.3 (issuing rbd commands to ceph cluster).

Gr. Stefan

-- 
| BIT BV  http://www.bit.nl/Kamer van Koophandel 09090351
| GPG: 0xD14839C6   +31 318 648 688 / i...@bit.nl
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[Konstantin Shalygin]

On 01/04/2018 11:38 PM, Stefan Kooman wrote:

Only luminous clients. Mostly rbd (qemu-kvm) images.


Who is managed your images? May be OpenStack Cinder?



k
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[Stefan Kooman]

Quoting Konstantin Shalygin (k0...@k0ste.ru):
> >This is still a pre-production cluster. Most tests have been done
> >using rbd. We did make some rbd clones / snapshots here and there.
> 
> What clients you used?

Only luminous clients. Mostly rbd (qemu-kvm) images.

Gr. Stefan

-- 
| BIT BV  http://www.bit.nl/Kamer van Koophandel 09090351
| GPG: 0xD14839C6   +31 318 648 688 / i...@bit.nl
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[ceph]

Am 3. Januar 2018 08:59:41 MEZ schrieb Stefan Kooman :
>Quoting Sage Weil (s...@newdream.net):
>> Hi Stefan, Mehmet,
>> 
>> Are these clusters that were upgraded from prior versions, or fresh 
>> luminous installs?
>
>Fresh luminous install... The cluster was installed with
>12.2.0, and later upgraded to 12.2.1 and 12.2.2.
>
In our Case we started with Jewel and then upgraded Step by Step to actual 
luminous  (12.2.2).

>> This message indicates that there is a stray clone object with no 
>> associated head or snapdir object.  That normally should never 
>> happen--it's presumably the result of a (hopefully old) bug.  The
>scrub 
>> process doesn't even clean them up, which maybe says something about
>how 
>> common it is/was...

I will have a look Next Week in my Logfiles  to see if it is still there...

>
>Hmm. This is still a pre-production cluster. Most tests have been done
>using rbd. We did make some rbd clones / snapshots here and there. We
>will
>re-install this cluster with a fresh 12.2.2 and try to reproduce.
>
>Gr. Stefan
 
Can not do a reinstall cause this is absolutely in production Environment :)

On Client side we using proxmox qemu/kvm  to Access the rbds  (librbd). 
Actually 12.2.2

- Mehmet 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[Konstantin Shalygin]

This is still a pre-production cluster. Most tests have been done
using rbd. We did make some rbd clones / snapshots here and there.


What clients you used?


k

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[Stefan Kooman]

Quoting Sage Weil (s...@newdream.net):
> Hi Stefan, Mehmet,
> 
> Are these clusters that were upgraded from prior versions, or fresh 
> luminous installs?

Fresh luminous install... The cluster was installed with
12.2.0, and later upgraded to 12.2.1 and 12.2.2.

> This message indicates that there is a stray clone object with no 
> associated head or snapdir object.  That normally should never 
> happen--it's presumably the result of a (hopefully old) bug.  The scrub 
> process doesn't even clean them up, which maybe says something about how 
> common it is/was...

Hmm. This is still a pre-production cluster. Most tests have been done
using rbd. We did make some rbd clones / snapshots here and there. We will
re-install this cluster with a fresh 12.2.2 and try to reproduce.

Gr. Stefan

-- 
| BIT BV  http://www.bit.nl/Kamer van Koophandel 09090351
| GPG: 0xD14839C6   +31 318 648 688 / i...@bit.nl
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[Sage Weil]

Hi Stefan, Mehmet,

Are these clusters that were upgraded from prior versions, or fresh 
luminous installs?

This message indicates that there is a stray clone object with no 
associated head or snapdir object.  That normally should never 
happen--it's presumably the result of a (hopefully old) bug.  The scrub 
process doesn't even clean them up, which maybe says something about how 
common it is/was...

sage


On Sun, 24 Dec 2017, c...@elchaka.de wrote:

> Hi Stefan,
> 
> Am 14. Dezember 2017 09:48:36 MEZ schrieb Stefan Kooman :
> >Hi,
> >
> >We see the following in the logs after we start a scrub for some osds:
> >
> >ceph-osd.2.log:2017-12-14 06:50:47.180344 7f0f47db2700  0
> >log_channel(cluster) log [DBG] : 1.2d8 scrub starts
> >ceph-osd.2.log:2017-12-14 06:50:47.180915 7f0f47db2700 -1 osd.2
> >pg_epoch: 11897 pg[1.2d8( v 11890'165209 (3221'163647,11890'165209]
> >local-lis/les=11733/11734 n=67 ec=132/132 lis/c 11733/11733 les/c/f
> >11734/11734/0 11733/11733/11733) [2,45,31] r=0 lpr=11733
> >crt=11890'165209 lcod 11890'165208 mlcod 11890'165208
> >active+clean+scrubbing] _scan_snaps no head for
> >1:1b518155:::rbd_data.620652ae8944a.0126:29 (have MIN)
> >ceph-osd.2.log:2017-12-14 06:50:47.180929 7f0f47db2700 -1 osd.2
> >pg_epoch: 11897 pg[1.2d8( v 11890'165209 (3221'163647,11890'165209]
> >local-lis/les=11733/11734 n=67 ec=132/132 lis/c 11733/11733 les/c/f
> >11734/11734/0 11733/11733/11733) [2,45,31] r=0 lpr=11733
> >crt=11890'165209 lcod 11890'165208 mlcod 11890'165208
> >active+clean+scrubbing] _scan_snaps no head for
> >1:1b518155:::rbd_data.620652ae8944a.0126:14 (have MIN)
> >ceph-osd.2.log:2017-12-14 06:50:47.180941 7f0f47db2700 -1 osd.2
> >pg_epoch: 11897 pg[1.2d8( v 11890'165209 (3221'163647,11890'165209]
> >local-lis/les=11733/11734 n=67 ec=132/132 lis/c 11733/11733 les/c/f
> >11734/11734/0 11733/11733/11733) [2,45,31] r=0 lpr=11733
> >crt=11890'165209 lcod 11890'165208 mlcod 11890'165208
> >active+clean+scrubbing] _scan_snaps no head for
> >1:1b518155:::rbd_data.620652ae8944a.0126:a (have MIN)
> >ceph-osd.2.log:2017-12-14 06:50:47.214198 7f0f43daa700  0
> >log_channel(cluster) log [DBG] : 1.2d8 scrub ok
> >
> >So finally it logs "scrub ok", but what does " _scan_snaps no head for
> >..." mean?
> 
> I also see this lines in our Logfiles and am wonder  what this means.
> 
> >Does this indicate a problem?
> 
> I do not guess so because we actually have not  any issues.
>  
> >
> >Ceph 12.2.2 with bluestore on lvm
> 
> We using 12.2.2 with filestore on xfs.
> 
> - Mehmet
> >
> >Gr. Stefan
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[Konstantin Shalygin]

So finally it logs "scrub ok", but what does " _scan_snaps no head for ..." 
mean?
Does this indicate a problem?

Ceph 12.2.2 with bluestore on lvm



I think this is because you have snaps created by client before 11.2.1. 
See http://tracker.ceph.com/issues/19413
I have already come across this on the Kraken (can't delete snapshot by 
not Jewel client). Because our Cinder client is Jewel.




k

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[ceph]

Hi Stefan,

Am 14. Dezember 2017 09:48:36 MEZ schrieb Stefan Kooman :
>Hi,
>
>We see the following in the logs after we start a scrub for some osds:
>
>ceph-osd.2.log:2017-12-14 06:50:47.180344 7f0f47db2700  0
>log_channel(cluster) log [DBG] : 1.2d8 scrub starts
>ceph-osd.2.log:2017-12-14 06:50:47.180915 7f0f47db2700 -1 osd.2
>pg_epoch: 11897 pg[1.2d8( v 11890'165209 (3221'163647,11890'165209]
>local-lis/les=11733/11734 n=67 ec=132/132 lis/c 11733/11733 les/c/f
>11734/11734/0 11733/11733/11733) [2,45,31] r=0 lpr=11733
>crt=11890'165209 lcod 11890'165208 mlcod 11890'165208
>active+clean+scrubbing] _scan_snaps no head for
>1:1b518155:::rbd_data.620652ae8944a.0126:29 (have MIN)
>ceph-osd.2.log:2017-12-14 06:50:47.180929 7f0f47db2700 -1 osd.2
>pg_epoch: 11897 pg[1.2d8( v 11890'165209 (3221'163647,11890'165209]
>local-lis/les=11733/11734 n=67 ec=132/132 lis/c 11733/11733 les/c/f
>11734/11734/0 11733/11733/11733) [2,45,31] r=0 lpr=11733
>crt=11890'165209 lcod 11890'165208 mlcod 11890'165208
>active+clean+scrubbing] _scan_snaps no head for
>1:1b518155:::rbd_data.620652ae8944a.0126:14 (have MIN)
>ceph-osd.2.log:2017-12-14 06:50:47.180941 7f0f47db2700 -1 osd.2
>pg_epoch: 11897 pg[1.2d8( v 11890'165209 (3221'163647,11890'165209]
>local-lis/les=11733/11734 n=67 ec=132/132 lis/c 11733/11733 les/c/f
>11734/11734/0 11733/11733/11733) [2,45,31] r=0 lpr=11733
>crt=11890'165209 lcod 11890'165208 mlcod 11890'165208
>active+clean+scrubbing] _scan_snaps no head for
>1:1b518155:::rbd_data.620652ae8944a.0126:a (have MIN)
>ceph-osd.2.log:2017-12-14 06:50:47.214198 7f0f43daa700  0
>log_channel(cluster) log [DBG] : 1.2d8 scrub ok
>
>So finally it logs "scrub ok", but what does " _scan_snaps no head for
>..." mean?

I also see this lines in our Logfiles and am wonder  what this means.

>Does this indicate a problem?

I do not guess so because we actually have not  any issues.
 
>
>Ceph 12.2.2 with bluestore on lvm

We using 12.2.2 with filestore on xfs.

- Mehmet
>
>Gr. Stefan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com