Re: [ceph-users] list admin issues

[Elias Abacioglu]

Maybe there are some advice here that can help remedy the situation a bit?
https://support.google.com/mail/answer/81126?hl=en
https://support.google.com/mail/answer/6227174?hl=en

/Elias

On Tue, Oct 9, 2018 at 2:24 AM Alex Gorbachev 
wrote:

> On Mon, Oct 8, 2018 at 7:48 AM Elias Abacioglu
>  wrote:
> >
> > If it's attachments causing this, perhaps forbid attachments? Force
> people to use pastebin / imgur type of services?
> >
> > /E
> >
> > On Mon, Oct 8, 2018 at 1:33 PM Martin Palma  wrote:
> >>
> >> Same here also on Gmail with G Suite.
> >> On Mon, Oct 8, 2018 at 12:31 AM Paul Emmerich 
> wrote:
> >> >
> >> > I'm also seeing this once every few months or so on Gmail with G
> Suite.
> >> >
> >> > Paul
> >> > Am So., 7. Okt. 2018 um 08:18 Uhr schrieb Joshua Chen
> >> > :
> >> > >
> >> > > I also got removed once, got another warning once (need to
> re-enable).
> >> > >
> >> > > Cheers
> >> > > Joshua
> >> > >
> >> > >
> >> > > On Sun, Oct 7, 2018 at 5:38 AM Svante Karlsson <
> svante.karls...@csi.se> wrote:
> >> > >>
> >> > >> I'm also getting removed but not only from ceph. I subscribe
> d...@kafka.apache.org list and the same thing happens there.
> >> > >>
> >> > >> Den lör 6 okt. 2018 kl 23:24 skrev Jeff Smith <
> j...@unbiasedgeek.com>:
> >> > >>>
> >> > >>> I have been removed twice.
> >> > >>> On Sat, Oct 6, 2018 at 7:07 AM Elias Abacioglu
> >> > >>>  wrote:
> >> > >>> >
> >> > >>> > Hi,
> >> > >>> >
> >> > >>> > I'm bumping this old thread cause it's getting annoying. My
> membership get disabled twice a month.
> >> > >>> > Between my two Gmail accounts I'm in more than 25 mailing lists
> and I see this behavior only here. Why is only ceph-users only affected?
> Maybe Christian was on to something, is this intentional?
> >> > >>> > Reality is that there is a lot of ceph-users with Gmail
> accounts, perhaps it wouldn't be so bad to actually trying to figure this
> one out?
> >> > >>> >
> >> > >>> > So can the maintainers of this list please investigate what
> actually gets bounced? Look at my address if you want.
> >> > >>> > I got disabled 20181006, 20180927, 20180916, 20180725, 20180718
> most recently.
> >> > >>> > Please help!
> >> > >>> >
> >> > >>> > Thanks,
> >> > >>> > Elias
> >> > >>> >
> >> > >>> > On Mon, Oct 16, 2017 at 5:41 AM Christian Balzer 
> wrote:
> >> > >>> >>
> >> > >>> >>
> >> > >>> >> Most mails to this ML score low or negatively with
> SpamAssassin, however
> >> > >>> >> once in a while (this is a recent one) we get relatively high
> scores.
> >> > >>> >> Note that the forged bits are false positives, but the SA is
> up to date and
> >> > >>> >> google will have similar checks:
> >> > >>> >> ---
> >> > >>> >> X-Spam-Status: No, score=3.9 required=10.0
> tests=BAYES_00,DCC_CHECK,
> >> > >>> >>
> FORGED_MUA_MOZILLA,FORGED_YAHOO_RCVD,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,
> >> > >>> >>
> HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MIME_HTML_MOSTLY,RCVD_IN_MSPIKE_H4,
> >> > >>> >>  RCVD_IN_MSPIKE_WL,RDNS_NONE,T_DKIM_INVALID shortcircuit=no
> autolearn=no
> >> > >>> >> ---
> >> > >>> >>
> >> > >>> >> Between attachment mails and some of these and you're well on
> your way out.
> >> > >>> >>
> >> > >>> >> The default mailman settings and logic require 5 bounces to
> trigger
> >> > >>> >> unsubscription and 7 days of NO bounces to reset the counter.
> >> > >>> >>
> >> > >>> >> Christian
> >> > >>> >>
> >> > >>> >> On Mon, 16 Oct 2017 12:23:25 +0900 Christian Balzer wrote:
> >> > >>> >>
> >> > >>> >> > On M

Re: [ceph-users] list admin issues

[Elias Abacioglu]

If it's attachments causing this, perhaps forbid attachments? Force people
to use pastebin / imgur type of services?

/E

On Mon, Oct 8, 2018 at 1:33 PM Martin Palma  wrote:

> Same here also on Gmail with G Suite.
> On Mon, Oct 8, 2018 at 12:31 AM Paul Emmerich 
> wrote:
> >
> > I'm also seeing this once every few months or so on Gmail with G Suite.
> >
> > Paul
> > Am So., 7. Okt. 2018 um 08:18 Uhr schrieb Joshua Chen
> > :
> > >
> > > I also got removed once, got another warning once (need to re-enable).
> > >
> > > Cheers
> > > Joshua
> > >
> > >
> > > On Sun, Oct 7, 2018 at 5:38 AM Svante Karlsson 
> wrote:
> > >>
> > >> I'm also getting removed but not only from ceph. I subscribe
> d...@kafka.apache.org list and the same thing happens there.
> > >>
> > >> Den lör 6 okt. 2018 kl 23:24 skrev Jeff Smith  >:
> > >>>
> > >>> I have been removed twice.
> > >>> On Sat, Oct 6, 2018 at 7:07 AM Elias Abacioglu
> > >>>  wrote:
> > >>> >
> > >>> > Hi,
> > >>> >
> > >>> > I'm bumping this old thread cause it's getting annoying. My
> membership get disabled twice a month.
> > >>> > Between my two Gmail accounts I'm in more than 25 mailing lists
> and I see this behavior only here. Why is only ceph-users only affected?
> Maybe Christian was on to something, is this intentional?
> > >>> > Reality is that there is a lot of ceph-users with Gmail accounts,
> perhaps it wouldn't be so bad to actually trying to figure this one out?
> > >>> >
> > >>> > So can the maintainers of this list please investigate what
> actually gets bounced? Look at my address if you want.
> > >>> > I got disabled 20181006, 20180927, 20180916, 20180725, 20180718
> most recently.
> > >>> > Please help!
> > >>> >
> > >>> > Thanks,
> > >>> > Elias
> > >>> >
> > >>> > On Mon, Oct 16, 2017 at 5:41 AM Christian Balzer 
> wrote:
> > >>> >>
> > >>> >>
> > >>> >> Most mails to this ML score low or negatively with SpamAssassin,
> however
> > >>> >> once in a while (this is a recent one) we get relatively high
> scores.
> > >>> >> Note that the forged bits are false positives, but the SA is up
> to date and
> > >>> >> google will have similar checks:
> > >>> >> ---
> > >>> >> X-Spam-Status: No, score=3.9 required=10.0
> tests=BAYES_00,DCC_CHECK,
> > >>> >>
> FORGED_MUA_MOZILLA,FORGED_YAHOO_RCVD,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,
> > >>> >>
> HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MIME_HTML_MOSTLY,RCVD_IN_MSPIKE_H4,
> > >>> >>  RCVD_IN_MSPIKE_WL,RDNS_NONE,T_DKIM_INVALID shortcircuit=no
> autolearn=no
> > >>> >> ---
> > >>> >>
> > >>> >> Between attachment mails and some of these and you're well on
> your way out.
> > >>> >>
> > >>> >> The default mailman settings and logic require 5 bounces to
> trigger
> > >>> >> unsubscription and 7 days of NO bounces to reset the counter.
> > >>> >>
> > >>> >> Christian
> > >>> >>
> > >>> >> On Mon, 16 Oct 2017 12:23:25 +0900 Christian Balzer wrote:
> > >>> >>
> > >>> >> > On Mon, 16 Oct 2017 14:15:22 +1100 Blair Bethwaite wrote:
> > >>> >> >
> > >>> >> > > Thanks Christian,
> > >>> >> > >
> > >>> >> > > You're no doubt on the right track, but I'd really like to
> figure out
> > >>> >> > > what it is at my end - I'm unlikely to be the only person
> subscribed
> > >>> >> > > to ceph-users via a gmail account.
> > >>> >> > >
> > >>> >> > > Re. attachments, I'm surprised mailman would be allowing them
> in the
> > >>> >> > > first place, and even so gmail's attachment requirements are
> less
> > >>> >> > > strict than most corporate email setups (those that don't
> already use
> > >>> >> > > a cloud provider).
> > >>> >> > >
> > >>> >> > M

Re: [ceph-users] list admin issues

[Elias Abacioglu]

Hi again,

Did some more investigating:
These are the mails I haven't received from yesterday and now:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-October/030357.html
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-October/030358.html
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-October/030359.html
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-October/030360.html

So it could be
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-October/030357.html
that bounced, and it has an attachment.

Does that particular mail have higher spam score?

/E

On Sat, Oct 6, 2018 at 3:06 PM Elias Abacioglu <
elias.abacio...@deltaprojects.com> wrote:

> Hi,
>
> I'm bumping this old thread cause it's getting annoying. My membership get
> disabled twice a month.
> Between my two Gmail accounts I'm in more than 25 mailing lists and I see
> this behavior only here. Why is only ceph-users only affected? Maybe
> Christian was on to something, is this intentional?
> Reality is that there is a lot of ceph-users with Gmail accounts, perhaps
> it wouldn't be so bad to actually trying to figure this one out?
>
> So can the maintainers of this list please investigate what actually gets
> bounced? Look at my address if you want.
> I got disabled 20181006, 20180927, 20180916, 20180725, 20180718 most
> recently.
> Please help!
>
> Thanks,
> Elias
>
> On Mon, Oct 16, 2017 at 5:41 AM Christian Balzer  wrote:
>
>>
>> Most mails to this ML score low or negatively with SpamAssassin, however
>> once in a while (this is a recent one) we get relatively high scores.
>> Note that the forged bits are false positives, but the SA is up to date
>> and
>> google will have similar checks:
>> ---
>> X-Spam-Status: No, score=3.9 required=10.0 tests=BAYES_00,DCC_CHECK,
>>
>>  
>> FORGED_MUA_MOZILLA,FORGED_YAHOO_RCVD,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,
>>
>>  
>> HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MIME_HTML_MOSTLY,RCVD_IN_MSPIKE_H4,
>>  RCVD_IN_MSPIKE_WL,RDNS_NONE,T_DKIM_INVALID shortcircuit=no autolearn=no
>> ---
>>
>> Between attachment mails and some of these and you're well on your way
>> out.
>>
>> The default mailman settings and logic require 5 bounces to trigger
>> unsubscription and 7 days of NO bounces to reset the counter.
>>
>> Christian
>>
>> On Mon, 16 Oct 2017 12:23:25 +0900 Christian Balzer wrote:
>>
>> > On Mon, 16 Oct 2017 14:15:22 +1100 Blair Bethwaite wrote:
>> >
>> > > Thanks Christian,
>> > >
>> > > You're no doubt on the right track, but I'd really like to figure out
>> > > what it is at my end - I'm unlikely to be the only person subscribed
>> > > to ceph-users via a gmail account.
>> > >
>> > > Re. attachments, I'm surprised mailman would be allowing them in the
>> > > first place, and even so gmail's attachment requirements are less
>> > > strict than most corporate email setups (those that don't already use
>> > > a cloud provider).
>> > >
>> > Mailman doesn't do anything with this by default AFAIK, but see below.
>> > Strict is fine if you're in control, corporate mail can be hell, doubly
>> so
>> > if on M$ cloud.
>> >
>> > > This started happening earlier in the year after I turned off digest
>> > > mode. I also have a paid google domain, maybe I'll try setting
>> > > delivery to that address and seeing if anything changes...
>> > >
>> > Don't think google domain is handled differently, but what do I know.
>> >
>> > Though the digest bit confirms my suspicion about attachments:
>> > ---
>> > When a subscriber chooses to receive plain text daily “digests” of list
>> > messages, Mailman sends the digest messages without any original
>> > attachments (in Mailman lingo, it “scrubs” the messages of attachments).
>> > However, Mailman also includes links to the original attachments that
>> the
>> > recipient can click on.
>> > ---
>> >
>> > Christian
>> >
>> > > Cheers,
>> > >
>> > > On 16 October 2017 at 13:54, Christian Balzer 
>> wrote:
>> > > >
>> > > > Hello,
>> > > >
>> > > > You're on gmail.
>> > > >
>> > > > Aside from various potential false positives with regards to spam
>> my bet
>> > > > is that gmail's known dislike for attachments is the cause of these
>> > > > bounces and that setting is beyond your control.

Re: [ceph-users] list admin issues

[Elias Abacioglu]

Hi,

I'm bumping this old thread cause it's getting annoying. My membership get
disabled twice a month.
Between my two Gmail accounts I'm in more than 25 mailing lists and I see
this behavior only here. Why is only ceph-users only affected? Maybe
Christian was on to something, is this intentional?
Reality is that there is a lot of ceph-users with Gmail accounts, perhaps
it wouldn't be so bad to actually trying to figure this one out?

So can the maintainers of this list please investigate what actually gets
bounced? Look at my address if you want.
I got disabled 20181006, 20180927, 20180916, 20180725, 20180718 most
recently.
Please help!

Thanks,
Elias

On Mon, Oct 16, 2017 at 5:41 AM Christian Balzer  wrote:

>
> Most mails to this ML score low or negatively with SpamAssassin, however
> once in a while (this is a recent one) we get relatively high scores.
> Note that the forged bits are false positives, but the SA is up to date and
> google will have similar checks:
> ---
> X-Spam-Status: No, score=3.9 required=10.0 tests=BAYES_00,DCC_CHECK,
>
>  
> FORGED_MUA_MOZILLA,FORGED_YAHOO_RCVD,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,
>
>  
> HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MIME_HTML_MOSTLY,RCVD_IN_MSPIKE_H4,
>  RCVD_IN_MSPIKE_WL,RDNS_NONE,T_DKIM_INVALID shortcircuit=no autolearn=no
> ---
>
> Between attachment mails and some of these and you're well on your way out.
>
> The default mailman settings and logic require 5 bounces to trigger
> unsubscription and 7 days of NO bounces to reset the counter.
>
> Christian
>
> On Mon, 16 Oct 2017 12:23:25 +0900 Christian Balzer wrote:
>
> > On Mon, 16 Oct 2017 14:15:22 +1100 Blair Bethwaite wrote:
> >
> > > Thanks Christian,
> > >
> > > You're no doubt on the right track, but I'd really like to figure out
> > > what it is at my end - I'm unlikely to be the only person subscribed
> > > to ceph-users via a gmail account.
> > >
> > > Re. attachments, I'm surprised mailman would be allowing them in the
> > > first place, and even so gmail's attachment requirements are less
> > > strict than most corporate email setups (those that don't already use
> > > a cloud provider).
> > >
> > Mailman doesn't do anything with this by default AFAIK, but see below.
> > Strict is fine if you're in control, corporate mail can be hell, doubly
> so
> > if on M$ cloud.
> >
> > > This started happening earlier in the year after I turned off digest
> > > mode. I also have a paid google domain, maybe I'll try setting
> > > delivery to that address and seeing if anything changes...
> > >
> > Don't think google domain is handled differently, but what do I know.
> >
> > Though the digest bit confirms my suspicion about attachments:
> > ---
> > When a subscriber chooses to receive plain text daily “digests” of list
> > messages, Mailman sends the digest messages without any original
> > attachments (in Mailman lingo, it “scrubs” the messages of attachments).
> > However, Mailman also includes links to the original attachments that the
> > recipient can click on.
> > ---
> >
> > Christian
> >
> > > Cheers,
> > >
> > > On 16 October 2017 at 13:54, Christian Balzer  wrote:
> > > >
> > > > Hello,
> > > >
> > > > You're on gmail.
> > > >
> > > > Aside from various potential false positives with regards to spam my
> bet
> > > > is that gmail's known dislike for attachments is the cause of these
> > > > bounces and that setting is beyond your control.
> > > >
> > > > Because Google knows best[tm].
> > > >
> > > > Christian
> > > >
> > > > On Mon, 16 Oct 2017 13:50:43 +1100 Blair Bethwaite wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> This is a mailing-list admin issue - I keep being unsubscribed from
> > > >> ceph-users with the message:
> > > >> "Your membership in the mailing list ceph-users has been disabled
> due
> > > >> to excessive bounces..."
> > > >> This seems to be happening on roughly a monthly basis.
> > > >>
> > > >> Thing is I have no idea what the bounce is or where it is coming
> from.
> > > >> I've tried emailing ceph-users-ow...@lists.ceph.com and the contact
> > > >> listed in Mailman (l...@redhat.com) to get more info but haven't
> > > >> received any response despite several attempts.
> > > >>
> > > >> Help!
> > > >>
> > > >
> > > >
> > > > --
> > > > Christian BalzerNetwork/Systems Engineer
> > > > ch...@gol.com   Rakuten Communications
> > >
> > >
> > >
> >
> >
>
>
> --
> Christian BalzerNetwork/Systems Engineer
> ch...@gol.com   Rakuten Communications
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] qemu/rbd: threads vs native, performance tuning

[Elias Abacioglu]

Hi,

I was reading this thread:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-March/008486.html

And I am trying to get better performance in my virtual machines.
These are my RBD settings:
"rbd_cache": "true",
"rbd_cache_block_writes_upfront": "false",
"rbd_cache_max_dirty": "25165824",
"rbd_cache_max_dirty_age": "1.00",
"rbd_cache_max_dirty_object": "0",
"rbd_cache_size": "33554432",
"rbd_cache_target_dirty": "16777216",
"rbd_cache_writethrough_until_flush": "true",

I decided to test native mode and ran fio like this inside a VM:
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test
--filename=random_read_write.fio --bs=4k --iodepth=64 --size=4G
--readwrite=randrw --rwmixread=75

I tested these two setups in qemu.



I ran fio a couple times to have a little variance and here is the results:

 READ: io=3071.7MB, aggrb=96718KB/s, minb=96718KB/s, maxb=96718KB/s,
mint=32521msec, maxt=32521msec
WRITE: io=1024.4MB, aggrb=32253KB/s, minb=32253KB/s, maxb=32253KB/s,
mint=32521msec, maxt=32521msec
 READ: io=3071.7MB, aggrb=96451KB/s, minb=96451KB/s, maxb=96451KB/s,
mint=32611msec, maxt=32611msec
WRITE: io=1024.4MB, aggrb=32164KB/s, minb=32164KB/s, maxb=32164KB/s,
mint=32611msec, maxt=32611msec
 READ: io=3071.7MB, aggrb=93763KB/s, minb=93763KB/s, maxb=93763KB/s,
mint=33546msec, maxt=33546msec
WRITE: io=1024.4MB, aggrb=31267KB/s, minb=31267KB/s, maxb=31267KB/s,
mint=33546msec, maxt=33546msec
---

DISK = [ driver = "raw" , cache = "directsync" , discard = "unmap" , io
= "native" ]
 READ: io=3071.7MB, aggrb=68771KB/s, minb=68771KB/s, maxb=68771KB/s,
mint=45737msec, maxt=45737msec
WRITE: io=1024.4MB, aggrb=22933KB/s, minb=22933KB/s, maxb=22933KB/s,
mint=45737msec, maxt=45737msec
 READ: io=3071.7MB, aggrb=67794KB/s, minb=67794KB/s, maxb=67794KB/s,
mint=46396msec, maxt=46396msec
WRITE: io=1024.4MB, aggrb=22607KB/s, minb=22607KB/s, maxb=22607KB/s,
mint=46396msec, maxt=46396msec
 READ: io=3071.7MB, aggrb=67536KB/s, minb=67536KB/s, maxb=67536KB/s,
mint=46573msec, maxt=46573msec
WRITE: io=1024.4MB, aggrb=22521KB/s, minb=22521KB/s, maxb=22521KB/s,
mint=46573msec, maxt=46573msec

So native is around 30-40% faster than threads according to this.
But I have a few questions now.
1. Is it safe to run cache='directsync' io='native' documentation refers to
writeback/threads?
2. How can I get even better performance? These benchmarks are from a pool
with 11 NVME Bluestore OSDs, 2x10Gb NIC. It feels pretty slow IMO.

Thanks,
Elias
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Why LZ4 isn't built with ceph?

[Elias Abacioglu]

Cool, then it's time to upgrade to Mimic.
Thanks for the info!


On Wed, Jul 25, 2018 at 6:37 PM, Casey Bodley  wrote:

>
> On 07/25/2018 08:39 AM, Elias Abacioglu wrote:
>
>> Hi
>>
>> I'm wondering why LZ4 isn't built by default for newer Linux distros like
>> Ubuntu Xenial?
>> I understand that it wasn't built for Trusty because of too old lz4
>> libraries. But why isn't built for the newer distros?
>>
>> Thanks,
>> Elias
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
> Hi Elias,
>
> We only turned it on by default once it was available on all target
> platforms, which wasn't the case until the mimic release. This happened in
> https://github.com/ceph/ceph/pull/21332, with some prior discussion in
> https://github.com/ceph/ceph/pull/17038.
>
> I don't know how to add build dependencies that are conditional on ubuntu
> version, but if you're keen to see this in luminous and have some debian
> packaging experience, you can target a PR against the luminous branch. I'm
> happy to help with review.
>
> Casey
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Why LZ4 isn't built with ceph?

[Elias Abacioglu]

Hi

I'm wondering why LZ4 isn't built by default for newer Linux distros like
Ubuntu Xenial?
I understand that it wasn't built for Trusty because of too old lz4
libraries. But why isn't built for the newer distros?

Thanks,
Elias
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Mon scrub errors

[Elias Abacioglu]

Hi,

Sorry for bumping an old thread, but I have the same issue.

Is it the mon.0 that I should reset?

2018-07-02 11:39:51.223731 [ERR]  mon.2 ScrubResult(keys
{monmap=48,osd_metadata=52} crc {monmap=3659766006,osd_metadata=1928984927})
2018-07-02 11:39:51.223701 [ERR]  mon.0 ScrubResult(keys
{monmap=45,osd_metadata=55} crc {monmap=1049715966,osd_metadata=409731371})
2018-07-02 11:39:51.223666 [ERR]  scrub mismatch
2018-07-02 11:39:51.223632 [ERR]  mon.1 ScrubResult(keys
{monmap=48,osd_metadata=52} crc {monmap=672606225,osd_metadata=1928984927})
2018-07-02 11:39:51.223598 [ERR]  mon.0 ScrubResult(keys
{monmap=45,osd_metadata=55} crc {monmap=1049715966,osd_metadata=409731371})
2018-07-02 11:39:51.223551 [ERR]  scrub mismatch
2018-07-02 11:39:51.211198 [ERR]  mon.2 ScrubResult(keys
{logm=33,mds_health=10,mds_metadata=1,mdsmap=56} crc
{logm=3564099269,mds_health=692648600,mds_metadata=2226414273,mdsmap=643765345})
2018-07-02 11:39:51.211148 [ERR]  mon.0 ScrubResult(keys
{logm=33,mds_health=6,mds_metadata=1,mdsmap=60} crc
{logm=3564099269,mds_health=4085298890,mds_metadata=2226414273,mdsmap=758287940})
2018-07-02 11:39:51.211100 [ERR]  scrub mismatch
2018-07-02 11:39:51.211052 [ERR]  mon.1 ScrubResult(keys
{logm=33,mds_health=9,mds_metadata=1,mdsmap=57} crc
{logm=3564099269,mds_health=4264808124,mds_metadata=2226414273,mdsmap=704105513})
2018-07-02 11:39:51.211000 [ERR]  mon.0 ScrubResult(keys
{logm=33,mds_health=6,mds_metadata=1,mdsmap=60} crc
{logm=3564099269,mds_health=4085298890,mds_metadata=2226414273,mdsmap=758287940})
2018-07-02 11:39:51.210944 [ERR]  scrub mismatch

2018-07-03 12:42:21.674471 [ERR]  mon.2 ScrubResult(keys
{monmap=24,osd_metadata=76} crc {monmap=578360729,osd_metadata=2641573038})
2018-07-03 12:42:21.674447 [ERR]  mon.0 ScrubResult(keys
{monmap=24,osd_metadata=76} crc {monmap=962305203,osd_metadata=2641573038})
2018-07-03 12:42:21.674422 [ERR]  scrub mismatch
2018-07-03 12:42:21.674399 [ERR]  mon.1 ScrubResult(keys
{monmap=24,osd_metadata=76} crc {monmap=3891180386,osd_metadata=2641573038})
2018-07-03 12:42:21.674359 [ERR]  mon.0 ScrubResult(keys
{monmap=24,osd_metadata=76} crc {monmap=962305203,osd_metadata=2641573038})
2018-07-03 12:42:21.674319 [ERR]  scrub mismatch
2018-07-03 12:42:21.664602 [ERR]  mon.2 ScrubResult(keys
{mgrstat=73,monmap=27} crc {mgrstat=199345228,monmap=2433199488})
2018-07-03 12:42:21.664523 [ERR]  mon.0 ScrubResult(keys
{mgrstat=73,monmap=27} crc {mgrstat=199345228,monmap=880923984})

2018-07-16 12:42:32.413856 [ERR]  mon.2 ScrubResult(keys
{monmap=17,osd_metadata=83} crc {monmap=3927456148,osd_metadata=4212506330})
2018-07-16 12:42:32.413828 [ERR]  mon.0 ScrubResult(keys
{monmap=17,osd_metadata=83} crc {monmap=4047155390,osd_metadata=4212506330})
2018-07-16 12:42:32.413800 [ERR]  scrub mismatch
2018-07-16 12:42:32.413769 [ERR]  mon.1 ScrubResult(keys
{monmap=17,osd_metadata=83} crc {monmap=797941615,osd_metadata=4212506330})
2018-07-16 12:42:32.413740 [ERR]  mon.0 ScrubResult(keys
{monmap=17,osd_metadata=83} crc {monmap=4047155390,osd_metadata=4212506330})
2018-07-16 12:42:32.413689 [ERR]  scrub mismatch
2018-07-16 12:42:32.412795 [ERR]  mon.2 ScrubResult(keys
{mgrstat=66,monmap=34} crc {mgrstat=1700428385,monmap=4227348033})
2018-07-16 12:42:32.412768 [ERR]  mon.0 ScrubResult(keys
{mgrstat=66,monmap=34} crc {mgrstat=1700428385,monmap=3150674595})
2018-07-16 12:42:32.412741 [ERR]  scrub mismatch


Thanks,
Elias

On Thu, Apr 5, 2018 at 4:40 PM, kefu chai  wrote:

> On Thu, Apr 5, 2018 at 3:10 PM, Rickard Nilsson
>  wrote:
> > Hi,
> >
> > Im' having a cluster with three moitors, two mds and nine osd. Lately
> I've
> > been getting scrub errors from the monitors;
> >
> > 2018-04-05 07:26:52.147185 [ERR]  mon.2 ScrubResult(keys
> > {osd_pg_creating=1,osdmap=99} crc
> > {osd_pg_creating=1404726104,osdmap=3323124730})
> > 2018-04-05 07:26:52.147167 [ERR]  mon.0 ScrubResult(keys
> > {osd_metadata=5,osd_pg_creating=1,osdmap=94} crc
> > {osd_metadata=477302505,osd_pg_creating=1404726104,osdmap=2387598890})
> > 2018-04-05 07:26:52.147139 [ERR]  scrub mismatch
> > 2018-04-05 07:26:52.144378 [ERR]  mon.2 ScrubResult(keys
> > {mgrstat=92,monmap=5,osd_pg_creating=1,osdmap=2} crc
> > {mgrstat=2630742218,monmap=4118007020,osd_pg_creating=
> 1404726104,osdmap=940126788})
> > 2018-04-05 07:26:52.144360 [ERR]  mon.0 ScrubResult(keys
> > {mgrstat=92,monmap=5,osd_metadata=3} crc
> > {mgrstat=2630742218,monmap=4118007020,osd_metadata=3256871745})
> > 2018-04-05 07:26:52.144334 [ERR]  scrub mismatch
> > 2018-04-05 07:26:52.140213 [ERR]  mon.2 ScrubResult(keys
> {mgr=67,mgrstat=33}
> > crc {mgr=1823433831,mgrstat=205032})
> > 2018-04-05 07:26:52.140193 [ERR]  mon.0 ScrubResult(keys
> > {mgr=67,mgr_command_descs=1,mgr_metadata=2,mgrstat=30} crc
> > {mgr=1823433831,mgr_command_descs=2758154725,mgr_metadata=
> 2776211204,mgrstat=779157107})
> > 2018-04-05 07:26:52.140165 [ERR]  scrub mismatch
> > 2018-04-05 07:26:52.120025 [ERR]  mon.2 ScrubResult(keys
> {logm=23,mdsmap=77}
> > 

Re: [ceph-users] upgrading jewel to luminous fails

[Elias Abacioglu]

Ok, I followed Michaels advice

I was able to start mon without deleting /var/lib/ceph. It forced me
to use the Ubuntu 16.04 image of ceph-container. The reason I wanted
to delete /var/lib/ceph was that I wanted to switch to the CentOS 7
image which is becoming more standard in ceph-container and probably
has better support.

Thanks
Elias

On Tue, Jun 19, 2018 at 10:16 AM, Elias Abacioglu
 wrote:
> Hi Mike
>
> On Mon, Jun 18, 2018 at 8:36 PM, Michael Kuriger  wrote:
>> If you delete /var/lib/ceph, all your authentication is gone.
>
> I'm runnig ceph-container with etcd as a K/V store. So it downloads
> the auth I believe. Perhaps I'm wrong.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] upgrading jewel to luminous fails

[Elias Abacioglu]

Hi Mike

On Mon, Jun 18, 2018 at 8:36 PM, Michael Kuriger  wrote:
> If you delete /var/lib/ceph, all your authentication is gone.

I'm runnig ceph-container with etcd as a K/V store. So it downloads
the auth I believe. Perhaps I'm wrong.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] upgrading jewel to luminous fails

[Elias Abacioglu]

Hi

I'm having some issues trying to upgrade jewel to luminous.
I've installed the new mon on one of the jewel mon nodes.
Before I start the luminous mon I remove /var/lib/ceph (cause i'm
running ceph-container and switching from the ubuntu image to centos
image).

The lum mon starts, but fails to join and then it causes one of the
jewel monitors to die.

Anyone got a clue on how I move forward from this?
I increased logging to 20 for mon and I've uploaded the logs here:
https://filedrop.deltaprojects.com/1e980a2c67ec
mon1 is the one being upgraded to luminous.

Thanks
Elias
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] what happen to the OSDs if the OS disk dies?

[Elias Abacioglu]

Hi Felix,

I have experience from running Ceph on SATADOM on R630. And it is kind of
bad cause we got bad SATADOM's from Dell.
If you are going to use SATADOM make sure to buy directly from a Innodisk
reseller and not from Dell.
We bought our SATADOM from Dell and they degraded in 5-6 months. And the
reason is that Dell is to cheap to get decent SATADOM, Innodisk got SATADOM
with and without TRIM. Dell resells the SATADOM's without TRIM and use them
for their Nutanix XC Series.

And here is the quirk. You won't find a regular 4-pin Molex power inside
the Dell R series, they have a small 4-pin power on the mobo next to the
internal SATA slots, but it's not a regular 4p 12v ATX, it is smaller.

So unless you can get Dell to sell custom power cables or better SATADOM,
you need to buy the SATADOM from Dell which includes a small cable that
fits and then you are screwed in a couple of months cause their SATADOM
doesn't do TRIM.

I've also tried using R630 internal USB port with Corsair Voyager GTX USB
flash drive which supports TRIM, but unfortunately Linux (we are running
v4.4.0) does not send TRIM over USB. In MS Windows TRIM works with that USB
drive, I tested that with my laptop using virtualbox.
So these USB drives will degrade as well.

Whenever you are trying to do something smart, there is always a quirk it
seems.

/Elias

On Tue, Aug 16, 2016 at 10:43 AM, Félix Barbeira 
wrote:

> Thanks everybody for the answers, it really helped me a lot. So, to sum
> up, this is the options that I have:
>
>
>- OS in a RAID1.
>   - PROS: the cluster is protected against OS failures. If one of
>   this disks fail, it could be easily replaced because it is 
> hot-swappable.
>   - CONS: we are "wasting" 2 bays of disks that could be destinated
>   to OSDs.
>
> * In the case of R730xd we have the option to put 2x2.5" SSDs disks on
> the slots on the back like Brian says. For me this is clearly the best
> option. We'll see if the department of finance has the same opinion :)
>
>
>- OS in a single disk.
>- PROS: we are using only 1 disk slot. It could be a cheaper disk than
>   the 4TB model because we are only going to use ~10GB.
>   - CONS: the OS is not protected against failures and if this disk
>   fails, the OSDs in this machine (11) fails too. In this case we might 
> try
>   to adjust the configuration in order to not reconstruct all this OSDs 
> data
>   and wait until the OS disk is replaced (I'm not sure if this is 
> possible, I
>   should check the docs).
>- OS in a SATADOM ( http://www.innodisk.com/intel/product.html )
>   - PROS: we have all the disk slots available to use for OSDs.
>   - CONS: I have no experience with this kind of devices, I'm not
>   sure if the are trustworthy. This devices are fast but they are not raid
>   protected, it's a single point of failure like the previous option.
>- OS boot from a SAN (this is the option I'm considering for the non
>R730xd machines, which does not have the 2x2.5" slots on the back).
>   - PROS: all the disk slots are available to OSDs. The OS disk is
>   protected by RAID on the remote storage.
>   - CONS: we depend of the network, I guess the OS device does not
>   require a lot of traffic, all the ceph OSDs network traffic should be
>   managed through another network card.
>
> Maybe I'm missing some other option, in that case please tell me, it would
> be helpful.
>
> It would be really helpful if somebody has experience with the option of
> booting OS from a SAN, sharing their pros/cons experience because that
> option it's very interesting to me.
>
>
> 2016-08-14 14:57 GMT+02:00 Christian Balzer :
>
>>
>> Hello,
>>
>> I shall top-quote, summarize here.
>>
>> Firstly we have to consider that Ceph is deployed by people with a wide
>> variety of needs, budgets and most of all cluster sizes.
>>
>> Wido has the pleasure (or is that nightmare? ^o^) to deal with a really
>> huge cluster, thousands of OSDs and an according larg number of nodes (if
>> memory serves me).
>>
>> While many others have comparatively small clusters, with decisively less
>> than 10 storage nodes, like me.
>>
>> So the approach and philosophy is obviously going to differ quite a bit
>> on either end of this spectrum.
>>
>> If you start large (dozens of nodes and hundreds of OSDs), where only a
>> small fraction of your data (10% or less) is in a failure domain (host
>> initially), then you can play fast and loose and save a lot of money by
>> designing your machines and infrastructure accordingly.
>> Things like redundant OS drives, PSUs, even network links on the host if
>> the cluster big enough.
>> In a cluster of sufficient size, a node failure and the resulting data
>> movements is just background noise.
>>
>> OTOH with smaller clusters, you obviously want to avoid failures if at all
>> possible, since not only the re-balancing is going to be more 

Re: [ceph-users] CephFS: mds client failing to respond to cache pressure

[Elias Abacioglu]

Hi Sean,

Isn't there any downsides to increasing the mds cache size?
My colleague mentioned that he tested it previously and then the cluster
didn't recover during a failover..

On Thu, Jun 9, 2016 at 12:41 PM, Sean Crosby <richardnixonsh...@gmail.com>
wrote:

> Hi Elias,
>
> When we have received the same warning, our solution has been to increase
> the inode cache on the MDS.
>
> We have added
>
> mds cache size = 200
>
>
> to the [global] section of ceph.conf on the MDS server. We have to restart
> MDS for the changes to be applied.
>
> Sean
>
>
> On 9 June 2016 at 19:55, Elias Abacioglu <
> elias.abacio...@deltaprojects.com> wrote:
>
>> Hi,
>>
>> I know this have been asked here a couple of times, but couldn't find
>> anything concrete.
>>
>> I have the following warning in our ceph cluster.
>> mds0: Client web01:cephfs.web01 failing to respond to cache pressure
>>
>> In previous Ceph versions this might have been a bug. But now we are
>> running Jewel.
>> So is there a way to fix this warning?
>> Do I need to tune some values? Boost the cluster? Boost the client?
>>
>> Here are some details:
>> Client kernel is 4.4.0.
>> Ceph 10.2.1
>>
>> # ceph mds dump
>> dumped fsmap epoch 5755
>> fs_namecephfs
>> epoch5755
>> flags0
>> created2015-12-03 11:21:28.128193
>> modified2016-05-16 06:48:47.969430
>> tableserver0
>> root0
>> session_timeout60
>> session_autoclose300
>> max_file_size1099511627776
>> last_failure4900
>> last_failure_osd_epoch5884
>> compatcompat={},rocompat={},incompat={1=base v0.20,2=client writeable
>> ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds
>> uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table}
>> max_mds1
>> in0
>> up{0=574261}
>> failed
>> damaged
>> stopped
>> data_pools2
>> metadata_pool3
>> inline_datadisabled
>> 574261:10.3.215.5:6801/62035 'ceph-mds03' mds.0.5609 up:active seq
>> 515014
>> 594257:10.3.215.10:6800/1386 'ceph-mds04' mds.0.0 up:standby-replay
>> seq 1
>>
>> Thanks,
>> Elias
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] CephFS: mds client failing to respond to cache pressure

[Elias Abacioglu]

Hi,

I know this have been asked here a couple of times, but couldn't find
anything concrete.

I have the following warning in our ceph cluster.
mds0: Client web01:cephfs.web01 failing to respond to cache pressure

In previous Ceph versions this might have been a bug. But now we are
running Jewel.
So is there a way to fix this warning?
Do I need to tune some values? Boost the cluster? Boost the client?

Here are some details:
Client kernel is 4.4.0.
Ceph 10.2.1

# ceph mds dump
dumped fsmap epoch 5755
fs_namecephfs
epoch5755
flags0
created2015-12-03 11:21:28.128193
modified2016-05-16 06:48:47.969430
tableserver0
root0
session_timeout60
session_autoclose300
max_file_size1099511627776
last_failure4900
last_failure_osd_epoch5884
compatcompat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds
uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table}
max_mds1
in0
up{0=574261}
failed
damaged
stopped
data_pools2
metadata_pool3
inline_datadisabled
574261:10.3.215.5:6801/62035 'ceph-mds03' mds.0.5609 up:active seq
515014
594257:10.3.215.10:6800/1386 'ceph-mds04' mds.0.0 up:standby-replay seq
1

Thanks,
Elias
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com