[ceph-users] leveldb to rocksdb migration

[Захаров Алексей]

Hi all.
I'm trying to change osd's kv backend using instructions mentioned here: 
http://pic.doit.com.cn/ceph/pdf/20180322/4/0401.pdf

But, ceph-osdomap-tool --check step fails with the following error:
ceph-osdomap-tool: /build/ceph-12.2.5/src/rocksdb/db/version_edit.h:188: void 
rocksdb::VersionEdit::AddFile(int, uint64_t, uint32_t, uint64_t, const 
rocksdb::InternalKey&, const rocksdb::InternalKey&, const SequenceNumber&, 
const SequenceNumber&, bool): Assertion `smallest_seqno <= largest_seqno' 
failed.
*** Caught signal (Aborted) **
 in thread 7f1b8d88e640 thread_name:ceph-osdomap-to
 ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous 
(stable)
 1: (()+0x189194) [0x55b3e66fa194]
 2: (()+0x11390) [0x7f1b84297390]
 3: (gsignal()+0x38) [0x7f1b83244428]
 4: (abort()+0x16a) [0x7f1b8324602a]
 5: (()+0x2dbd7) [0x7f1b8323cbd7]
 6: (()+0x2dc82) [0x7f1b8323cc82]
 7: (rocksdb::VersionSet::WriteSnapshot(rocksdb::log::Writer*)+0xad0) 
[0x55b3e675abc0]
 8: (rocksdb::VersionSet::LogAndApply(rocksdb::ColumnFamilyData*, 
rocksdb::MutableCFOptions const&, rocksdb::autovector const&, rocksdb::InstrumentedMutex*, rocksdb::Directory*, bool, 
rocksdb::ColumnFamilyOptions const*)+0x1624) [0x55b3e6764524]
 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector > const&, unsigned long*, bool)+0x1c48) 
[0x55b3e672d438]
 10: (rocksdb::DBImpl::Recover(std::vector const&, bool, bool, 
bool)+0x8c4) [0x55b3e672e544]
 11: (rocksdb::DB::Open(rocksdb::DBOptions const&, 
std::__cxx11::basic_string 
const&, std::vector const&, 
std::vector >*, rocksdb::DB**)+0xedc) 
[0x55b3e672f90c]
 12: (rocksdb::DB::Open(rocksdb::Options const&, 
std::__cxx11::basic_string 
const&, rocksdb::DB**)+0x6b1) [0x55b3e67311a1]
 13: (RocksDBStore::do_open(std::ostream&, bool)+0x89c) [0x55b3e66e4adc]
 14: (main()+0xa55) [0x55b3e6638ea5]
 15: (__libc_start_main()+0xf0) [0x7f1b8322f830]
 16: (_start()+0x29) [0x55b3e66b73b9]
Aborted

i've found this issue https://github.com/facebook/rocksdb/issues/946 , but i 
don't see easy workaround for it.
Has anyone faced the same problem?
Is there a tool to add fields to manifest files? For example to add 
largest_seqno. Or, is there a way to ignore this assert?

Using: ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous 
(stable)
Updated from jewel 10.2.10.

-- 
Regards,
Aleksei Zakharov

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cache tier

[Захаров Алексей]

Hi,
We use write-around cache tier with libradosstriper-based clients. We faced 
with bug which causes performance degradation: 
http://tracker.ceph.com/issues/22528 . Especially if it is a lot of small 
objects - sizeof(1 striper chunk). Such objects will promote on every 
read/write lock:).
And it is very hard to benchmark cache tier.

Also, we have a little testing pool with rbd disks for vm's. It works better 
with cache tier on ssd's. But, there's no heavy i/o load.

It's better to benchmark cache tier for your special case and choose cache mode 
based on benchmark results.

06.03.2018, 02:28, "Budai Laszlo" :
> Dear all,
>
> I have some questions about cache tier in ceph:
>
> 1. Can someone share experiences with cache tiering? What are the sensitive 
> things to pay attention regarding the cache tier? Can one use the same ssd 
> for both cache and
> 2. Is cache tiering supported with bluestore? Any advices for using cache 
> tier with Bluestore?
>
> Kind regards,
> Laszlo
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-- 
Regards,
Aleksei Zakharov

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cache tier unexpected behavior: promote on lock

[Захаров Алексей]

Thanks for the answers!
As it leads to a decrease of caching efficiency, i've opened an issue:
http://tracker.ceph.com/issues/22528 

15.12.2017, 23:03, "Gregory Farnum" <gfar...@redhat.com>:
> On Thu, Dec 14, 2017 at 9:11 AM, Захаров Алексей <zakharov@yandex.ru> 
> wrote:
>>  Hi, Gregory,
>>  Thank you for your answer!
>>
>>  Is there a way to not promote on "locking", when not using EC pools?
>>  Is it possible to make this configurable?
>>
>>  We don't use EC pool. So, for us this meachanism is overhead. It only adds
>>  more load on both pools and network.
>
> Unfortunately I don't think there's an easy way to avoid it that
> exists right now. The caching is generally not set up well for
> handling these kinds of things, but it's possible the logic to proxy
> class operations onto replicated pools might not be *too*
> objectionable
> -Greg
>
>>  14.12.2017, 01:16, "Gregory Farnum" <gfar...@redhat.com>:
>>
>>  Voluntary “locking” in RADOS is an “object class” operation. These are not
>>  part of the core API and cannot run on EC pools, so any operation using them
>>  will cause an immediate promotion.
>>  On Wed, Dec 13, 2017 at 4:02 AM Захаров Алексей <zakharov@yandex.ru>
>>  wrote:
>>
>>  Hello,
>>
>>  I've found that when client gets lock on object then ceph ignores any
>>  promotion settings and promotes this object immedeatly.
>>
>>  Is it a bug or a feature?
>>  Is it configurable?
>>
>>  Hope for any help!
>>
>>  Ceph version: 10.2.10 and 12.2.2
>>  We use libradosstriper-based clients.
>>
>>  Cache pool settings:
>>  size: 3
>>  min_size: 2
>>  crash_replay_interval: 0
>>  pg_num: 2048
>>  pgp_num: 2048
>>  crush_ruleset: 0
>>  hashpspool: true
>>  nodelete: false
>>  nopgchange: false
>>  nosizechange: false
>>  write_fadvise_dontneed: false
>>  noscrub: true
>>  nodeep-scrub: false
>>  hit_set_type: bloom
>>  hit_set_period: 60
>>  hit_set_count: 30
>>  hit_set_fpp: 0.05
>>  use_gmt_hitset: 1
>>  auid: 0
>>  target_max_objects: 0
>>  target_max_bytes: 18819770744832
>>  cache_target_dirty_ratio: 0.4
>>  cache_target_dirty_high_ratio: 0.6
>>  cache_target_full_ratio: 0.8
>>  cache_min_flush_age: 60
>>  cache_min_evict_age: 180
>>  min_read_recency_for_promote: 15
>>  min_write_recency_for_promote: 15
>>  fast_read: 0
>>  hit_set_grade_decay_rate: 50
>>  hit_set_search_last_n: 30
>>
>>  To get lock via cli (to test behavior) we use:
>>  # rados -p poolname lock get --lock-tag weird_ceph_locks --lock-cookie
>>  `uuid` objectname striper.lock
>>  Right after that object could be found in caching pool.
>>
>>  --
>>  Regards,
>>  Aleksei Zakharov
>>  ___
>>  ceph-users mailing list
>>  ceph-users@lists.ceph.com
>>  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>  --
>>  Regards,
>>  Aleksei Zakharov

-- 
Regards,
Aleksei Zakharov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cache tier unexpected behavior: promote on lock

[Захаров Алексей]

Hi, Gregory,Thank you for your answer! Is there a way to not promote on "locking", when not using EC pools?Is it possible to make this configurable? We don't use EC pool. So, for us this meachanism is overhead. It only adds more load on both pools and network. 14.12.2017, 01:16, "Gregory Farnum" <gfar...@redhat.com>:Voluntary “locking” in RADOS is an “object class” operation. These are not part of the core API and cannot run on EC pools, so any operation using them will cause an immediate promotion.On Wed, Dec 13, 2017 at 4:02 AM Захаров Алексей <zakharov@yandex.ru> wrote:Hello,I've found that when client gets lock on object then ceph ignores any promotion settings and promotes this object immedeatly.Is it a bug or a feature?Is it configurable?Hope for any help!Ceph version: 10.2.10 and 12.2.2We use libradosstriper-based clients.Cache pool settings:size: 3min_size: 2crash_replay_interval: 0pg_num: 2048pgp_num: 2048crush_ruleset: 0hashpspool: truenodelete: falsenopgchange: falsenosizechange: falsewrite_fadvise_dontneed: falsenoscrub: truenodeep-scrub: falsehit_set_type: bloomhit_set_period: 60hit_set_count: 30hit_set_fpp: 0.05use_gmt_hitset: 1auid: 0target_max_objects: 0target_max_bytes: 18819770744832cache_target_dirty_ratio: 0.4cache_target_dirty_high_ratio: 0.6cache_target_full_ratio: 0.8cache_min_flush_age: 60cache_min_evict_age: 180min_read_recency_for_promote: 15min_write_recency_for_promote: 15fast_read: 0hit_set_grade_decay_rate: 50hit_set_search_last_n: 30To get lock via cli (to test behavior) we use:# rados -p poolname lock get --lock-tag weird_ceph_locks --lock-cookie `uuid` objectname striper.lockRight after that object could be found in caching pool.--Regards,Aleksei Zakharov___ceph-users mailing listceph-users@lists.ceph.comhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com  -- Regards,Aleksei Zakharov ___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Cache tier unexpected behavior: promote on lock

[Захаров Алексей]

Hello,

I've found that when client gets lock on object then ceph ignores any promotion 
settings and promotes this object immedeatly.

Is it a bug or a feature?
Is it configurable?

Hope for any help!

Ceph version: 10.2.10 and 12.2.2
We use libradosstriper-based clients.

Cache pool settings:
size: 3
min_size: 2
crash_replay_interval: 0
pg_num: 2048
pgp_num: 2048
crush_ruleset: 0
hashpspool: true
nodelete: false
nopgchange: false
nosizechange: false
write_fadvise_dontneed: false
noscrub: true
nodeep-scrub: false
hit_set_type: bloom
hit_set_period: 60
hit_set_count: 30
hit_set_fpp: 0.05
use_gmt_hitset: 1
auid: 0
target_max_objects: 0
target_max_bytes: 18819770744832
cache_target_dirty_ratio: 0.4
cache_target_dirty_high_ratio: 0.6
cache_target_full_ratio: 0.8
cache_min_flush_age: 60
cache_min_evict_age: 180
min_read_recency_for_promote: 15
min_write_recency_for_promote: 15
fast_read: 0
hit_set_grade_decay_rate: 50
hit_set_search_last_n: 30

To get lock via cli (to test behavior) we use:
# rados -p poolname lock get --lock-tag weird_ceph_locks --lock-cookie `uuid` 
objectname striper.lock
Right after that object could be found in caching pool.


-- 
Regards,
Aleksei Zakharov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Recovery operations and ioprio options

[Захаров Алексей]

Hi, Nick
Thank you for the answer!

It's still unclear for me, do those options have no effect at all?
Or disk thread is used for some other operations?

09.11.2017, 04:18, "Nick Fisk" :
>>  -Original Message-
>>  From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
>>  ??? ???
>>  Sent: 08 November 2017 16:21
>>  To: ceph-users@lists.ceph.com
>>  Subject: [ceph-users] Recovery operations and ioprio options
>>
>>  Hello,
>>  Today we use ceph jewel with:
>>    osd disk thread ioprio class=idle
>>    osd disk thread ioprio priority=7
>>  and "nodeep-scrub" flag is set.
>>
>>  We want to change scheduler from CFQ to deadline, so these options will
>>  lose effect.
>>  I've tried to find out what operations are performed in "disk thread".
>
> What I
>>  found is that only scrubbing and snap-trimming operations are performed in
>>  "disk thread".
>
> In jewel those operations are now in the main OSD thread and setting the
> ioprio's will have no effect. Use the scrub and snap trim sleep options to
> throttle them.
>
>>  Do these options affect recovery operations?
>>  Are there any other operations in "disk thread", except scrubbing and
>
> snap-
>>  trimming?
>>
>>  --
>>  Regards,
>>  Aleksei Zakharov
>>  ___
>>  ceph-users mailing list
>>  ceph-users@lists.ceph.com
>>  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-- 
Regards,
Aleksei Zakharov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Recovery operations and ioprio options

[Захаров Алексей]

Hello,
Today we use ceph jewel with:
  osd disk thread ioprio class=idle
  osd disk thread ioprio priority=7
and "nodeep-scrub" flag is set.

We want to change scheduler from CFQ to deadline, so these options will lose 
effect.
I've tried to find out what operations are performed in "disk thread". What I 
found is that only scrubbing and snap-trimming operations are performed in 
"disk thread".

Do these options affect recovery operations?
Are there any other operations in "disk thread", except scrubbing and 
snap-trimming?

-- 
Regards,
Aleksei Zakharov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com