subject:"\[ceph\-users\] Active\+clean PGs reported many times in log"

Re: [ceph-users] Active+clean PGs reported many times in log

2017-11-20 Thread Gregory Farnum

Is this from a time when it was displaying the doubled active+clean
outputs? Otherwise you'll need to retrieve a specific map when it was.
I'll ask around if anybody's seen this before; Jewel has been out a
while and the pg output like this changed dramatically for Luminous so
it may not be an issue in the latest LTS.
-Greg

On Mon, Nov 20, 2017 at 7:02 PM, Matteo Dacrema  wrote:
> I was running 10.2.7 but I’ve upgraded to 10.2.10 few days ago.
>
> Here Pg dump:
>
> https://owncloud.enter.it/index.php/s/AaD5Fc5tA6c8i1G
>
>
>
> Il giorno 19 nov 2017, alle ore 11:15, Gregory Farnum 
> ha scritto:
>
> On Tue, Nov 14, 2017 at 1:09 AM Matteo Dacrema  wrote:
>>
>> Hi,
>> I noticed that sometimes the monitors start to log active+clean pgs many
>> times in the same line. For example I have 18432 and the logs shows " 2136
>> active+clean, 28 active+clean, 2 active+clean+scrubbing+deep, 16266
>> active+clean;”
>> After a minute monitor start to log correctly again.
>>
>>
>> Is it normal ?
>
>
> That definitely looks weird to me, but I can imagine a few ways for it to
> occur. What version of Ceph are you running? Can you extract the pgmap and
> post the binary somewhere?
>
>>
>>
>> 2017-11-13 11:05:08.876724 7fb35d17d700  0 log_channel(cluster) log [INF]
>> : pgmap v99797105: 18432 pgs: 3 active+clean+scrubbing+deep, 18429
>> active+clean; 59520 GB data, 129 TB used, 110 TB / 239 TB avail; 40596 kB/s
>> rd, 89723 kB/s wr, 4899 op/s
>> 2017-11-13 11:05:09.911266 7fb35d17d700  0 log_channel(cluster) log [INF]
>> : pgmap v99797106: 18432 pgs: 2 active+clean+scrubbing+deep, 18430
>> active+clean; 59520 GB data, 129 TB used, 110 TB / 239 TB avail; 45931 kB/s
>> rd, 114 MB/s wr, 6179 op/s
>> 2017-11-13 11:05:10.751378 7fb359cfb700  0 mon.controller001@0(leader) e1
>> handle_command mon_command({"prefix": "osd pool stats", "format": "json"} v
>> 0) v1
>> 2017-11-13 11:05:10.751599 7fb359cfb700  0 log_channel(audit) log [DBG] :
>> from='client.? MailScanner warning: numerical links are often malicious:
>> 10.16.24.127:0/547552484' entity='client.telegraf' cmd=[{"prefix": "osd pool
>> stats", "format": "json"}]: dispatch
>> 2017-11-13 11:05:10.926839 7fb35d17d700  0 log_channel(cluster) log [INF]
>> : pgmap v99797107: 18432 pgs: 3 active+clean+scrubbing+deep, 18429
>> active+clean; 59520 GB data, 129 TB used, 110 TB / 239 TB avail; 47617 kB/s
>> rd, 134 MB/s wr, 7414 op/s
>> 2017-11-13 11:05:11.921115 7fb35d17d700  1 mon.controller001@0(leader).osd
>> e120942 e120942: 216 osds: 216 up, 216 in
>> 2017-11-13 11:05:11.926818 7fb35d17d700  0 log_channel(cluster) log [INF]
>> : osdmap e120942: 216 osds: 216 up, 216 in
>> 2017-11-13 11:05:11.984732 7fb35d17d700  0 log_channel(cluster) log [INF]
>> : pgmap v99797109: 18432 pgs: 3 active+clean+scrubbing+deep, 18429
>> active+clean; 59520 GB data, 129 TB used, 110 TB / 239 TB avail; 54110 kB/s
>> rd, 115 MB/s wr, 7827 op/s
>> 2017-11-13 11:05:13.085799 7fb35d17d700  0 log_channel(cluster) log [INF]
>> : pgmap v99797110: 18432 pgs: 973 active+clean, 12 active+clean, 3
>> active+clean+scrubbing+deep, 17444 active+clean; 59520 GB data, 129 TB used,
>> 110 TB / 239 TB avail; 115 MB/s rd, 90498 kB/s wr, 8490 op/s
>> 2017-11-13 11:05:14.181219 7fb35d17d700  0 log_channel(cluster) log [INF]
>> : pgmap v99797111: 18432 pgs: 2136 active+clean, 28 active+clean, 2
>> active+clean+scrubbing+deep, 16266 active+clean; 59520 GB data, 129 TB used,
>> 110 TB / 239 TB avail; 136 MB/s rd, 94461 kB/s wr, 10237 op/s
>> 2017-11-13 11:05:15.324630 7fb35d17d700  0 log_channel(cluster) log [INF]
>> : pgmap v99797112: 18432 pgs: 3179 active+clean, 44 active+clean, 2
>> active+clean+scrubbing+deep, 15207 active+clean; 59519 GB data, 129 TB used,
>> 110 TB / 239 TB avail; 184 MB/s rd, 81743 kB/s wr, 13786 op/s
>> 2017-11-13 11:05:16.381452 7fb35d17d700  0 log_channel(cluster) log [INF]
>> : pgmap v99797113: 18432 pgs: 3600 active+clean, 52 active+clean, 2
>> active+clean+scrubbing+deep, 14778 active+clean; 59518 GB data, 129 TB used,
>> 110 TB / 239 TB avail; 208 MB/s rd, 77342 kB/s wr, 14382 op/s
>> 2017-11-13 11:05:17.272757 7fb3570f2700  1 leveldb: Level-0 table
>> #26314650: started
>> 2017-11-13 11:05:17.390808 7fb3570f2700  1 leveldb: Level-0 table
>> #26314650: 18281928 bytes OK
>> 2017-11-13 11:05:17.392636 7fb3570f2700  1 leveldb: Delete type=0
>> #26314647
>>
>> 2017-11-13 11:05:17.397516 7fb3570f2700  1 leveldb: Manual compaction at
>> level-0 from 'pgmap\x0099796362' @ 72057594037927935 : 1 ..
>> 'pgmap\x0099796613' @ 0 : 0; will stop at 'pgmap_pg\x006.ff' @ 29468156273 :
>> 1
>>
>>
>> Thank you
>> Matteo
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> --
> Questo messaggio e' stato analizzato con Libra ESVA ed e' risultato non
> infetto.
> Clicca qui per segnalarlo come spam.
> Clicca qui per metterlo in blacklist
>

Re: [ceph-users] Active+clean PGs reported many times in log

2017-11-20 Thread Matteo Dacrema

I was running 10.2.7 but I’ve upgraded to 10.2.10 few days ago.

Here Pg dump:

https://owncloud.enter.it/index.php/s/AaD5Fc5tA6c8i1G 




> Il giorno 19 nov 2017, alle ore 11:15, Gregory Farnum  ha 
> scritto:
> 
> On Tue, Nov 14, 2017 at 1:09 AM Matteo Dacrema  > wrote:
> Hi,
> I noticed that sometimes the monitors start to log active+clean pgs many 
> times in the same line. For example I have 18432 and the logs shows " 2136 
> active+clean, 28 active+clean, 2 active+clean+scrubbing+deep, 16266 
> active+clean;”
> After a minute monitor start to log correctly again.
> 
> Is it normal ?
> 
> That definitely looks weird to me, but I can imagine a few ways for it to 
> occur. What version of Ceph are you running? Can you extract the pgmap and 
> post the binary somewhere?
>  
> 
> 2017-11-13 11:05:08.876724 7fb35d17d700  0 log_channel(cluster) log [INF] : 
> pgmap v99797105: 18432 pgs: 3 active+clean+scrubbing+deep, 18429 
> active+clean; 59520 GB data, 129 TB used, 110 TB / 239 TB avail; 40596 kB/s 
> rd, 89723 kB/s wr, 4899 op/s
> 2017-11-13 11:05:09.911266 7fb35d17d700  0 log_channel(cluster) log [INF] : 
> pgmap v99797106: 18432 pgs: 2 active+clean+scrubbing+deep, 18430 
> active+clean; 59520 GB data, 129 TB used, 110 TB / 239 TB avail; 45931 kB/s 
> rd, 114 MB/s wr, 6179 op/s
> 2017-11-13 11:05:10.751378 7fb359cfb700  0 mon.controller001@0(leader) e1 
> handle_command mon_command({"prefix": "osd pool stats", "format": "json"} v 
> 0) v1
> 2017-11-13 11:05:10.751599 7fb359cfb700  0 log_channel(audit) log [DBG] : 
> from='client.? MailScanner warning: numerical links are often malicious: 
> 10.16.24.127:0/547552484 ' 
> entity='client.telegraf' cmd=[{"prefix": "osd pool stats", "format": 
> "json"}]: dispatch
> 2017-11-13 11:05:10.926839 7fb35d17d700  0 log_channel(cluster) log [INF] : 
> pgmap v99797107: 18432 pgs: 3 active+clean+scrubbing+deep, 18429 
> active+clean; 59520 GB data, 129 TB used, 110 TB / 239 TB avail; 47617 kB/s 
> rd, 134 MB/s wr, 7414 op/s
> 2017-11-13 11:05:11.921115 7fb35d17d700  1 mon.controller001@0(leader).osd 
> e120942 e120942: 216 osds: 216 up, 216 in
> 2017-11-13 11:05:11.926818 7fb35d17d700  0 log_channel(cluster) log [INF] : 
> osdmap e120942: 216 osds: 216 up, 216 in
> 2017-11-13 11:05:11.984732 7fb35d17d700  0 log_channel(cluster) log [INF] : 
> pgmap v99797109: 18432 pgs: 3 active+clean+scrubbing+deep, 18429 
> active+clean; 59520 GB data, 129 TB used, 110 TB / 239 TB avail; 54110 kB/s 
> rd, 115 MB/s wr, 7827 op/s
> 2017-11-13 11:05:13.085799 7fb35d17d700  0 log_channel(cluster) log [INF] : 
> pgmap v99797110: 18432 pgs: 973 active+clean, 12 active+clean, 3 
> active+clean+scrubbing+deep, 17444 active+clean; 59520 GB data, 129 TB used, 
> 110 TB / 239 TB avail; 115 MB/s rd, 90498 kB/s wr, 8490 op/s
> 2017-11-13 11:05:14.181219 7fb35d17d700  0 log_channel(cluster) log [INF] : 
> pgmap v99797111: 18432 pgs: 2136 active+clean, 28 active+clean, 2 
> active+clean+scrubbing+deep, 16266 active+clean; 59520 GB data, 129 TB used, 
> 110 TB / 239 TB avail; 136 MB/s rd, 94461 kB/s wr, 10237 op/s
> 2017-11-13 11:05:15.324630 7fb35d17d700  0 log_channel(cluster) log [INF] : 
> pgmap v99797112: 18432 pgs: 3179 active+clean, 44 active+clean, 2 
> active+clean+scrubbing+deep, 15207 active+clean; 59519 GB data, 129 TB used, 
> 110 TB / 239 TB avail; 184 MB/s rd, 81743 kB/s wr, 13786 op/s
> 2017-11-13 11:05:16.381452 7fb35d17d700  0 log_channel(cluster) log [INF] : 
> pgmap v99797113: 18432 pgs: 3600 active+clean, 52 active+clean, 2 
> active+clean+scrubbing+deep, 14778 active+clean; 59518 GB data, 129 TB used, 
> 110 TB / 239 TB avail; 208 MB/s rd, 77342 kB/s wr, 14382 op/s
> 2017-11-13 11:05:17.272757 7fb3570f2700  1 leveldb: Level-0 table #26314650: 
> started
> 2017-11-13 11:05:17.390808 7fb3570f2700  1 leveldb: Level-0 table #26314650: 
> 18281928 bytes OK
> 2017-11-13 11:05:17.392636 7fb3570f2700  1 leveldb: Delete type=0 #26314647
> 
> 2017-11-13 11:05:17.397516 7fb3570f2700  1 leveldb: Manual compaction at 
> level-0 from 'pgmap\x0099796362' @ 72057594037927935 : 1 .. 
> 'pgmap\x0099796613' @ 0 : 0; will stop at 'pgmap_pg\x006.ff' @ 29468156273 : 1
> 
> 
> Thank you
> Matteo
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
> 
> 
> -- 
> Questo messaggio e' stato analizzato con Libra ESVA ed e' risultato non 
> infetto. 
> Clicca qui per segnalarlo come spam. 
>  
> Clicca qui per metterlo in blacklist 
> 
___
ceph-users mailing list

Re: [ceph-users] Active+clean PGs reported many times in log

2017-11-19 Thread Gregory Farnum

On Tue, Nov 14, 2017 at 1:09 AM Matteo Dacrema  wrote:

> Hi,
> I noticed that sometimes the monitors start to log active+clean pgs many
> times in the same line. For example I have 18432 and the logs shows " 2136
> active+clean, 28 active+clean, 2 active+clean+scrubbing+deep, 16266
> active+clean;”
> After a minute monitor start to log correctly again.


> Is it normal ?
>

That definitely looks weird to me, but I can imagine a few ways for it to
occur. What version of Ceph are you running? Can you extract the pgmap and
post the binary somewhere?


>
> 2017-11-13 11:05:08.876724 7fb35d17d700  0 log_channel(cluster) log [INF]
> : pgmap v99797105: 18432 pgs: 3 active+clean+scrubbing+deep, 18429
> active+clean; 59520 GB data, 129 TB used, 110 TB / 239 TB avail; 40596 kB/s
> rd, 89723 kB/s wr, 4899 op/s
> 2017-11-13 11:05:09.911266 7fb35d17d700  0 log_channel(cluster) log [INF]
> : pgmap v99797106: 18432 pgs: 2 active+clean+scrubbing+deep, 18430
> active+clean; 59520 GB data, 129 TB used, 110 TB / 239 TB avail; 45931 kB/s
> rd, 114 MB/s wr, 6179 op/s
> 2017-11-13 11:05:10.751378 7fb359cfb700  0 mon.controller001@0(leader) e1
> handle_command mon_command({"prefix": "osd pool stats", "format": "json"} v
> 0) v1
> 2017-11-13 11:05:10.751599 7fb359cfb700  0 log_channel(audit) log [DBG] :
> from='client.? 10.16.24.127:0/547552484' entity='client.telegraf'
> cmd=[{"prefix": "osd pool stats", "format": "json"}]: dispatch
> 2017-11-13 11:05:10.926839 7fb35d17d700  0 log_channel(cluster) log [INF]
> : pgmap v99797107: 18432 pgs: 3 active+clean+scrubbing+deep, 18429
> active+clean; 59520 GB data, 129 TB used, 110 TB / 239 TB avail; 47617 kB/s
> rd, 134 MB/s wr, 7414 op/s
> 2017-11-13 11:05:11.921115 7fb35d17d700  1 mon.controller001@0(leader).osd
> e120942 e120942: 216 osds: 216 up, 216 in
> 2017-11-13 11:05:11.926818 7fb35d17d700  0 log_channel(cluster) log [INF]
> : osdmap e120942: 216 osds: 216 up, 216 in
> 2017-11-13 11:05:11.984732 7fb35d17d700  0 log_channel(cluster) log [INF]
> : pgmap v99797109: 18432 pgs: 3 active+clean+scrubbing+deep, 18429
> active+clean; 59520 GB data, 129 TB used, 110 TB / 239 TB avail; 54110 kB/s
> rd, 115 MB/s wr, 7827 op/s
> 2017-11-13 11:05:13.085799 7fb35d17d700  0 log_channel(cluster) log [INF]
> : pgmap v99797110: 18432 pgs: 973 active+clean, 12 active+clean, 3
> active+clean+scrubbing+deep, 17444 active+clean; 59520 GB data, 129 TB
> used, 110 TB / 239 TB avail; 115 MB/s rd, 90498 kB/s wr, 8490 op/s
> 2017-11-13 11:05:14.181219 7fb35d17d700  0 log_channel(cluster) log [INF]
> : pgmap v99797111: 18432 pgs: 2136 active+clean, 28 active+clean, 2
> active+clean+scrubbing+deep, 16266 active+clean; 59520 GB data, 129 TB
> used, 110 TB / 239 TB avail; 136 MB/s rd, 94461 kB/s wr, 10237 op/s
> 2017-11-13 11:05:15.324630 7fb35d17d700  0 log_channel(cluster) log [INF]
> : pgmap v99797112: 18432 pgs: 3179 active+clean, 44 active+clean, 2
> active+clean+scrubbing+deep, 15207 active+clean; 59519 GB data, 129 TB
> used, 110 TB / 239 TB avail; 184 MB/s rd, 81743 kB/s wr, 13786 op/s
> 2017-11-13 11:05:16.381452 7fb35d17d700  0 log_channel(cluster) log [INF]
> : pgmap v99797113: 18432 pgs: 3600 active+clean, 52 active+clean, 2
> active+clean+scrubbing+deep, 14778 active+clean; 59518 GB data, 129 TB
> used, 110 TB / 239 TB avail; 208 MB/s rd, 77342 kB/s wr, 14382 op/s
> 2017-11-13 11:05:17.272757 7fb3570f2700  1 leveldb: Level-0 table
> #26314650: started
> 2017-11-13 11:05:17.390808 7fb3570f2700  1 leveldb: Level-0 table
> #26314650: 18281928 bytes OK
> 2017-11-13 11:05:17.392636 7fb3570f2700  1 leveldb: Delete type=0 #26314647
>
> 2017-11-13 11:05:17.397516 7fb3570f2700  1 leveldb: Manual compaction at
> level-0 from 'pgmap\x0099796362' @ 72057594037927935 : 1 ..
> 'pgmap\x0099796613' @ 0 : 0; will stop at 'pgmap_pg\x006.ff' @ 29468156273
> : 1
>
>
> Thank you
> Matteo
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Active+clean PGs reported many times in log

2017-11-13 Thread Matteo Dacrema

Hi, 
I noticed that sometimes the monitors start to log active+clean pgs many times 
in the same line. For example I have 18432 and the logs shows " 2136 
active+clean, 28 active+clean, 2 active+clean+scrubbing+deep, 16266 
active+clean;”
After a minute monitor start to log correctly again.

Is it normal ?

2017-11-13 11:05:08.876724 7fb35d17d700  0 log_channel(cluster) log [INF] : 
pgmap v99797105: 18432 pgs: 3 active+clean+scrubbing+deep, 18429 active+clean; 
59520 GB data, 129 TB used, 110 TB / 239 TB avail; 40596 kB/s rd, 89723 kB/s 
wr, 4899 op/s
2017-11-13 11:05:09.911266 7fb35d17d700  0 log_channel(cluster) log [INF] : 
pgmap v99797106: 18432 pgs: 2 active+clean+scrubbing+deep, 18430 active+clean; 
59520 GB data, 129 TB used, 110 TB / 239 TB avail; 45931 kB/s rd, 114 MB/s wr, 
6179 op/s
2017-11-13 11:05:10.751378 7fb359cfb700  0 mon.controller001@0(leader) e1 
handle_command mon_command({"prefix": "osd pool stats", "format": "json"} v 0) 
v1
2017-11-13 11:05:10.751599 7fb359cfb700  0 log_channel(audit) log [DBG] : 
from='client.? 10.16.24.127:0/547552484' entity='client.telegraf' 
cmd=[{"prefix": "osd pool stats", "format": "json"}]: dispatch
2017-11-13 11:05:10.926839 7fb35d17d700  0 log_channel(cluster) log [INF] : 
pgmap v99797107: 18432 pgs: 3 active+clean+scrubbing+deep, 18429 active+clean; 
59520 GB data, 129 TB used, 110 TB / 239 TB avail; 47617 kB/s rd, 134 MB/s wr, 
7414 op/s
2017-11-13 11:05:11.921115 7fb35d17d700  1 mon.controller001@0(leader).osd 
e120942 e120942: 216 osds: 216 up, 216 in
2017-11-13 11:05:11.926818 7fb35d17d700  0 log_channel(cluster) log [INF] : 
osdmap e120942: 216 osds: 216 up, 216 in
2017-11-13 11:05:11.984732 7fb35d17d700  0 log_channel(cluster) log [INF] : 
pgmap v99797109: 18432 pgs: 3 active+clean+scrubbing+deep, 18429 active+clean; 
59520 GB data, 129 TB used, 110 TB / 239 TB avail; 54110 kB/s rd, 115 MB/s wr, 
7827 op/s
2017-11-13 11:05:13.085799 7fb35d17d700  0 log_channel(cluster) log [INF] : 
pgmap v99797110: 18432 pgs: 973 active+clean, 12 active+clean, 3 
active+clean+scrubbing+deep, 17444 active+clean; 59520 GB data, 129 TB used, 
110 TB / 239 TB avail; 115 MB/s rd, 90498 kB/s wr, 8490 op/s
2017-11-13 11:05:14.181219 7fb35d17d700  0 log_channel(cluster) log [INF] : 
pgmap v99797111: 18432 pgs: 2136 active+clean, 28 active+clean, 2 
active+clean+scrubbing+deep, 16266 active+clean; 59520 GB data, 129 TB used, 
110 TB / 239 TB avail; 136 MB/s rd, 94461 kB/s wr, 10237 op/s
2017-11-13 11:05:15.324630 7fb35d17d700  0 log_channel(cluster) log [INF] : 
pgmap v99797112: 18432 pgs: 3179 active+clean, 44 active+clean, 2 
active+clean+scrubbing+deep, 15207 active+clean; 59519 GB data, 129 TB used, 
110 TB / 239 TB avail; 184 MB/s rd, 81743 kB/s wr, 13786 op/s
2017-11-13 11:05:16.381452 7fb35d17d700  0 log_channel(cluster) log [INF] : 
pgmap v99797113: 18432 pgs: 3600 active+clean, 52 active+clean, 2 
active+clean+scrubbing+deep, 14778 active+clean; 59518 GB data, 129 TB used, 
110 TB / 239 TB avail; 208 MB/s rd, 77342 kB/s wr, 14382 op/s
2017-11-13 11:05:17.272757 7fb3570f2700  1 leveldb: Level-0 table #26314650: 
started
2017-11-13 11:05:17.390808 7fb3570f2700  1 leveldb: Level-0 table #26314650: 
18281928 bytes OK
2017-11-13 11:05:17.392636 7fb3570f2700  1 leveldb: Delete type=0 #26314647

2017-11-13 11:05:17.397516 7fb3570f2700  1 leveldb: Manual compaction at 
level-0 from 'pgmap\x0099796362' @ 72057594037927935 : 1 .. 'pgmap\x0099796613' 
@ 0 : 0; will stop at 'pgmap_pg\x006.ff' @ 29468156273 : 1


Thank you
Matteo

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Active+clean PGs reported many times in log

Re: [ceph-users] Active+clean PGs reported many times in log

Re: [ceph-users] Active+clean PGs reported many times in log

[ceph-users] Active+clean PGs reported many times in log

4 matches

Site Navigation

Mail list logo

Footer information