Re: [ceph-users] Unsolved questions

2017-02-06 Thread Konstantin Shalygin
1) Every once in a while, some processes (PHP) accessing the filesystem get stuck in a D-state (Uninterruptable sleep). I wonder if this happens due to network fluctuations (both server are connected via a simple Gigabit crosslink cable) or how to diagnose this. Why exactly does this happen in

Re: [ceph-users] cephfs and erasure coding

2017-03-29 Thread Konstantin Shalygin
a lot of locking, opening en closing those small files which is not very efficient. http://tracker.ceph.com/issues/12430 That is in development right now. NO code yet out there, but should be there later this year. -- Best regards, Konstantin Shalygin

Re: [ceph-users] cephfs and erasure coding

2017-03-29 Thread Konstantin Shalygin
don't. -- Best regards, Konstantin Shalygin ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] cephfs and erasure coding

2017-03-29 Thread Konstantin Shalygin
Hello. How your tests? I'm looking for CephFS with EC for save space on replicas for many small files (dovecot mailboxes). Thanks everyone for the input. We are online in our test environment and are running user workflows to make sure everything is running as expected.

Re: [ceph-users] Changing pg_num on cache pool

2017-05-27 Thread Konstantin Shalygin
# ceph osd pool set cephfs_data_cache pg_num 256 Error EPERM: splits in cache pools must be followed by scrubs and leave sufficient free space to avoid overfilling. use --yes-i-really-mean-it to force. Is there something I need to do, before increasing PGs on a cache pool? Can this be

Re: [ceph-users] Changing pg_num on cache pool

2017-05-28 Thread Konstantin Shalygin
You can also just remove the caching from the pool, increase the pgs, then set it back up as a cache pool. It'll require downtime if it's in front of an EC rbd pool or EC cephfs on Jewel or Hammer, but it won't take long as all of the objects will be gone. Why do you need to increase the

Re: [ceph-users] Changing pg_num on cache pool

2017-05-28 Thread Konstantin Shalygin
On 05/28/2017 09:43 PM, David Turner wrote: What are your pg numbers for each pool? Your % used in each pool? And number of OSDs? GLOBAL: SIZE AVAIL RAW USED %RAW USED 89380G 74755G 14625G 16.36 POOLS: NAME ID USED

Re: [ceph-users] rocksdb: Corruption: missing start of fragmented record

2017-11-12 Thread Konstantin Shalygin
Fair point. I just tried with 12.2.1 (on pre-release Ubuntu bionic now). Doesn't change anything - fsck doesn't fix rocksdb, the bluestore won't mount, the OSD won't activate and the error is the same. Is there any fix in .2 that might address this, or do you just mean that in general there

Re: [ceph-users] question pool usage vs. pool usage raw

2017-11-23 Thread Konstantin Shalygin
The way is: "rbd help disk-usage" -- Best regards, Konstantin Shalygin ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Journal / WAL drive size?

2017-11-23 Thread Konstantin Shalygin
When I look at the drive partitions the DB / WAL partitions are only 576Mb & 1GB respectively. This feels a bit small. Before deploy bluestore OSD you should define your db/wal size via: ceph_bluestore_block_db_size: '32212254720' # 30Gb ceph_bluestore_block_wal_size: '1073741824' # 1Gb

Re: [ceph-users] question pool usage vs. pool usage raw

2017-11-23 Thread Konstantin Shalygin
What is the difference between the "usage" and the "raw usage" of a pool? Usage - is your data. Raw - is what actually your data use with all copies (pool 'size' option). I.e. if your data is 1000G - your raw is 3000G. ___ ceph-users mailing list

Re: [ceph-users] Problems understanding 'ceph features' output

2017-12-16 Thread Konstantin Shalygin
So I assume there are 3 ceph applications (e.g. three VMs) on the jewel host, and 5 applications on the two luminous hosts. To be clear - client is not VM, client is disk. If one VM have 3 disks - 'ceph features' show 3 clients. k ___ ceph-users

Re: [ceph-users] [Luminous 12.2.2] Cluster peformance drops after certain point of time

2017-12-17 Thread Konstantin Shalygin
I am testing luminous 12.2.2 and find a strange behavior of my cluster. Check your block.db usage. Luminous 12.2.2 is affected http://tracker.ceph.com/issues/22264 [root@ceph-osd0]# ceph daemon osd.46 perf dump | jq '.bluefs' | grep -E '(db|slow)'   "db_total_bytes": 30064762880,  

[ceph-users] ceph.com/logos: luminous missed.

2017-12-13 Thread Konstantin Shalygin
Luminous logo is absent. I saw here http://ceph.com/releases/v12-2-0-luminous-released/ luminous logo exists, but this one is very low resolution. k ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] determining the source of io in the cluster

2017-12-18 Thread Konstantin Shalygin
I can see, that the io/read ops come from the pool where we store VM volumes, but i can't source this issue to a particular volume. You can use this script https://github.com/cernceph/ceph-scripts/blob/master/tools/rbd-io-stats.pl This is for filestore only. I adapted it to use bluestore for

Re: [ceph-users] 1 osd Segmentation fault in test cluster

2017-12-14 Thread Konstantin Shalygin
>/Is this useful for someone? / Yes! Seehttp://tracker.ceph.com/issues/21259 The latest luminous branch (which you can get from https://shaman.ceph.com/builds/ceph/luminous/) has some additional debugging on OSD shutdown that should help me figure out what is causing this. If this is

[ceph-users] Is the 12.2.1 really stable? Anybody have production cluster with Luminous Bluestore?

2017-11-16 Thread Konstantin Shalygin
Hi cephers. Some thoughts... At this time my cluster on Kraken 11.2.0 - works smooth with FileStore and RBD only. I want upgrade to Luminous 12.2.1 and go to Bluestore because this cluster want grows double with new disks, so is best opportunity migrate to Bluestore. In ML I was found two

Re: [ceph-users] rocksdb: Corruption: missing start of fragmented record

2017-11-13 Thread Konstantin Shalygin
Which isn't released yet, yes. I could try building the development repository if you think that has a chance of resolving the issue? For tests - yes... This ML tells that 12.2.2 should be based on commit

Re: [ceph-users] features required for live migration

2017-11-13 Thread Konstantin Shalygin
I'd like to use the live migration feature of KVM. In this scenario, what features may be enabled in the rbd base image? and in my EV (snapshot clone)? You can use live migration without features. For KVM I can recommend minimal "rbd default features = 3" (layering, striping).

Re: [ceph-users] features required for live migration

2017-11-14 Thread Konstantin Shalygin
d features? -- Best regards, Konstantin Shalygin ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] features required for live migration

2017-11-14 Thread Konstantin Shalygin
t;yum upgrade -y" in order to update all operating system components. And please, apollogize me but In your lines I am not able to find the answer to my questions. Please, can you clarify? -- Best regards, Konstantin Shalygin ___ ceph-users mailing

Re: [ceph-users] features required for live migration

2017-11-14 Thread Konstantin Shalygin
- you can set minimal features (like '3') - this is enough for virtualization (snapshots, clones). And start your project. -- Best regards, Konstantin Shalygin ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph

Re: [ceph-users] Is the 12.2.1 really stable? Anybody have production cluster with Luminous Bluestore?

2017-11-16 Thread Konstantin Shalygin
We upgraded from firefly to 12.2.1 You still on FileStore? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Is the 12.2.1 really stable? Anybody have production cluster with Luminous Bluestore?

2017-11-16 Thread Konstantin Shalygin
My cluster (55 OSDs) runs 12.2.x since the release, and bluestore too All good so far This is cleanly deployed cluster or upgrade from some version? ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] Is the 12.2.1 really stable? Anybody have production cluster with Luminous Bluestore?

2017-11-16 Thread Konstantin Shalygin
I was told at the Openstack Summit that 12.2.2 should drop "In a few days." That was a week ago yesterday. If you have a little leeway, it may be best to wait. I know I am, but I'm paranoid. There was also a performance regression mentioned recently that's supposed to be fixed. As we can see

Re: [ceph-users] oVirt/RHEV and Ceph

2017-11-01 Thread Konstantin Shalygin
OpenStack database backup once at hour/day. There <http://lists.ovirt.org/pipermail/users/2017-October/084796.html>, on ovirt-users mail list Matthias Leopold perform first tests with Luminous. I'm on the way to Luminous, too. -- Best regards, Kon

Re: [ceph-users] Is the 12.2.1 really stable? Anybody have production cluster with Luminous Bluestore?

2017-12-03 Thread Konstantin Shalygin
Hi, We're running 12.2.1 on production and facing some memory & cpu issues --> http://tracker.ceph.com/issues/4?next_issue_id=3_issue_id=5 http://tracker.ceph.com/issues/21933 Try 12.2.2 http://ceph.com/releases/v12-2-2-luminous-released/

Re: [ceph-users] Luminous 12.2.2 rpm's not signed?

2017-12-04 Thread Konstantin Shalygin
Total size: 51 M Is this ok [y/d/N]: y Downloading packages: Package ceph-common-12.2.2-0.el7.x86_64.rpm is not signed http://tracker.ceph.com/issues/22311 ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] Hangs with qemu/libvirt/rbd when one host disappears

2017-12-07 Thread Konstantin Shalygin
the rule of thumb is 3 for small to mid-sized cluster. 3 mons works with 1+ OSD with Luminous: http://ceph.com/community/new-luminous-scalability/ ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] Hangs with qemu/libvirt/rbd when one host disappears

2017-12-07 Thread Konstantin Shalygin
as the 1Gb network is completely busy in such a scenario i would assume maybe the problem is that some network communication got stuck somewhere 1Gbit is nothing for ceph OSD hosts. Even if you use only spinners. Don't forget 1Gbit is much more latency and less speed (obviously) compared with

Re: [ceph-users] Memory leak in OSDs running 12.2.1 beyond the buffer_anon mempool leak

2017-12-05 Thread Konstantin Shalygin
We are trying out Ceph on a small cluster and are observing memory leakage in the OSD processes. Try new 12.2.2 - this release should fix memory issues with Bluestore. ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] Removing a ceph node and ceph documentation.

2017-12-09 Thread Konstantin Shalygin
Hi, 1. Is there a clear procedure documented for removing a failed ceph-node ? I have read and performed removing ODSs successfully, but are there commands that can be used to remove the node itself ? 2. On the ceph documentation website, how can I find Jewel, Kraken or Luminous specific docs ?

Re: [ceph-users] The way to minimize osd memory usage?

2017-12-09 Thread Konstantin Shalygin
I am testing running ceph luminous(12.2.1-249-g42172a4 (42172a443183ffe6b36e85770e53fe678db293bf) on ARM server. Try new 12.2.2 - this release should fix memory issues with Bluestore. ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] Ceph disk failure causing outage/ stalled writes

2017-12-20 Thread Konstantin Shalygin
he writes to the cluster gets stalled for a good 10-15 mins and actually the machine just shut itself down. If your writes is stalled due host shutdown than your cluster have wrong configuration - like 2 osd-hosts with min_size 2 and failure domain - host. As is usually difficult to give

Re: [ceph-users] POOL_NEARFULL

2017-12-21 Thread Konstantin Shalygin
Update your ceph.conf file This is also not help. I was create ticket http://tracker.ceph.com/issues/22520 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Proper way of removing osds

2017-12-21 Thread Konstantin Shalygin
Is this the correct way to removes OSDs, or am I doing something wrong ? Generic way for maintenance (e.g. disk replace) is rebalance by change osd weight: ceph osd crush reweight osdid 0 cluster migrate data "from this osd" When HEALTH_OK you can safe remove this OSD: ceph osd out osd_id

Re: [ceph-users] Show and Tell: Grafana cluster dashboard

2018-05-07 Thread Konstantin Shalygin
And a question: Is there a way to get the Cluster IOPS with prometheus metrics? I did this with collectd, but can't find a suitable metric from ceph-mgr. sum(irate(ceph_pool_rd[30s])) sum(irate(ceph_pool_wr[30s])) k ___ ceph-users mailing list

Re: [ceph-users] cluster can't remapped objects after change crush tree

2018-04-27 Thread Konstantin Shalygin
On 04/27/2018 05:05 PM, Igor Gajsin wrote: I have a crush rule like You still can use device classes! * host0 has a piece of data on osd.0 Not peace, full object. If we talk about non-EC pools. * host1 has pieces of data on osd.1 and osd.2 host1 has copy on osd.1 *or* osd.2 * host2 has

Re: [ceph-users] cluster can't remapped objects after change crush tree

2018-04-27 Thread Konstantin Shalygin
On 04/27/2018 04:37 PM, Igor Gajsin wrote: pool 7 'rbd' replicated size 3 min_size 2 crush_rule 0 Your pools have proper size settings - is 3. But you crush have only 2 buckets for this rule (e.g. is your pods). For making this rule work you should have minimum of 3 'pod' buckets. k

Re: [ceph-users] cluster can't remapped objects after change crush tree

2018-04-27 Thread Konstantin Shalygin
On 04/26/2018 11:30 PM, Igor Gajsin wrote: after assigning this rule to a pool it stucks in the same state: `ceph osd pool ls detail` please k ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] Collecting BlueStore per Object DB overhead

2018-04-27 Thread Konstantin Shalygin
I've writting a piece of Python code which can be run on a server running OSDs and will print the overhead. https://gist.github.com/wido/b1328dd45aae07c45cb8075a24de9f1f Feedback on this script is welcome, but also the output of what people are observing. For mixed (filestore / bluestore)

Re: [ceph-users] List pgs waiting to scrub?

2018-05-13 Thread Konstantin Shalygin
I hope I'm not asking something obvious (my google-fu wasn't strong enough), but how do I list how many pgs are scheduled for a (deep) scrub? Actually scrub is infinity process by design. How often your scrubs is ruled by scrub settings. You can list deep scrub time stamps to figure out what

Re: [ceph-users] which kernel support object-map, fast-diff

2018-05-15 Thread Konstantin Shalygin
So which kernel version support those feature? No one kernel support this features yet. k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW won't start after upgrade to 12.2.5

2018-05-21 Thread Konstantin Shalygin
The default configuration for rgw_ldap_secret seems to be set to /etc/openldap/secret, which on my system is empty: Please, create issue on tracker . Thanks. k ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] ceph-disk is getting removed from master

2018-05-25 Thread Konstantin Shalygin
ceph-disk should be considered as "frozen" and deprecated for Mimic, in favor of ceph-volume. ceph-volume will continue to support bare block device, i.e. without lvm'ish stuff? k ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] RBD features and feature journaling performance

2018-05-16 Thread Konstantin Shalygin
I'm trying to better understand rbd features but I have only found the information on the RBD page, is there any further RBD feature information and implementation? http://tracker.ceph.com/issues/15000 k ___ ceph-users mailing list

Re: [ceph-users] Why the change from ceph-disk to ceph-volume and lvm? (and just not stick with direct disk access)

2018-06-08 Thread Konstantin Shalygin
- ceph-disk was replaced for two reasons: (1) It's design was centered around udev, and it was terrible. We have been plagued for years with bugs due to race conditions in the udev-driven activation of OSDs, mostly variations of "I rebooted and not all of my OSDs started." It's horrible to

Re: [ceph-users] Why the change from ceph-disk to ceph-volume and lvm? (and just not stick with direct disk access)

2018-06-08 Thread Konstantin Shalygin
http://docs.ceph.com/docs/master/ceph-volume/simple/ ? Only 'scan' & 'activate'. Not 'create'. k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Why the change from ceph-disk to ceph-volume and lvm? (and just not stick with direct disk access)

2018-06-08 Thread Konstantin Shalygin
What is the reasoning behind switching to lvm? Does it make sense to go through (yet) another layer to access the disk? Why creating this dependency and added complexity? It is fine as it is, or not? In fact, the question is why one tool is replaced by another without saving functionality.

Re: [ceph-users] PM1633a

2018-06-16 Thread Konstantin Shalygin
Hello List - anyone using these drives and have any good / bad things to say about them? A few moths ago I was asking about PM1725 http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-February/024950.html No feedback, so I bought HGST SN260, because on the same price is better for

Re: [ceph-users] Add ssd's to hdd cluster, crush map class hdd update necessary?

2018-06-12 Thread Konstantin Shalygin
Is it necessary to update the crush map with class hdd Before adding ssd's the cluster? Of course, if this osds of one root. It is not necessary to manually edit crush: ceph osd crush rule create-replicated replicated_hosts_hdd default host hdd ceph osd crush rule create-replicated

Re: [ceph-users] OSDs too slow to start

2018-06-12 Thread Konstantin Shalygin
Each node now has 1 SSD with the OS and the BlockDBs and 3 HDDs with bluestore data. Very. Very bad idea. When your ssd/nvme dead you lost your linux box. k ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] Add ssd's to hdd cluster, crush map class hdd update necessary?

2018-06-13 Thread Konstantin Shalygin
On 06/13/2018 09:01 AM, Marc Roos wrote: Yes but I already have some sort of test cluster with data in it. I don’t think there are commands to modify existing rules that are being used by pools. And the default replicated_ruleset doesn’t have a class specified. I also have an erasure code rule

Re: [ceph-users] *****SPAM***** Re: Add ssd's to hdd cluster, crush map class hdd update necessary?

2018-06-13 Thread Konstantin Shalygin
On 06/13/2018 12:06 PM, Marc Roos wrote: Shit, I added this class and now everything start backfilling (10%) How is this possible, I only have hdd's? This is normal when you change your crush and placement rules. Post your output, I will take a look ceph osd crush tree ceph osd crush dump

Re: [ceph-users] OSDs too slow to start

2018-06-13 Thread Konstantin Shalygin
On 06/13/2018 08:22 PM, Alfredo Daniel Rezinovsky wrote: I have 3 boxes. And I'm installing a new one. Any box can be lost without data problem. If any SSD is lost I will just reinstall the whole box, still have data duplicates and in about 40 hours the triplicates will be ready. I

Re: [ceph-users] ceph-disk is getting removed from master

2018-05-30 Thread Konstantin Shalygin
On 05/30/2018 07:26 PM, Alfredo Deza wrote: If you don't want LVM, you can continue to use ceph-disk. How I can do this if ceph-disk will be removed from master? I really don't understand: why we need LV for new osds. k ___ ceph-users mailing

Re: [ceph-users] ceph-disk is getting removed from master

2018-05-30 Thread Konstantin Shalygin
On 05/30/2018 07:08 PM, Alfredo Deza wrote: ceph-volume accepts a bare block device as input, but it will create an LV behind the scenes I think this is regression. What if I don't need LV? k ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] Backfill stops after a while after OSD reweight

2018-06-22 Thread Konstantin Shalygin
Yes, I know that section of the docs, but can't find how to change the crush rules after "ceph osd crush tunables ...". Could you give me a hint? What you mean? All what you need after upgrade to Luminous is: ceph osd crush tunables optimal ceph osd crush set-all-straw-buckets-to-straw2 k

Re: [ceph-users] separate monitoring node

2018-06-20 Thread Konstantin Shalygin
Hi, at the moment, we use Icinga2, check_ceph* and Telegraf with the Ceph plugin. I'm asking what I need to have a separate host, which knows all about the Ceph cluster health. The reason is, that each OSD node has mostly the exact same data, which is transmitted into our database (like InfluxDB

Re: [ceph-users] Ceph Luminous RocksDB vs WalDB?

2018-06-26 Thread Konstantin Shalygin
I am playing with Ceph Luminous and getting confused information around usage of WalDB vs RocksDB. I have 2TB NVMe drive which I want to use for Wal/Rocks DB and have 5 2TB SSD's for OSD. I am planning to create 5 30GB partitions for RocksDB on NVMe drive, do I need to create partitions of WalDB

Re: [ceph-users] cluster can't remapped objects after change crush tree

2018-04-25 Thread Konstantin Shalygin
# ceph osd crush tree ID CLASS WEIGHT TYPE NAME -1 3.63835 root default -9 0.90959 pod group1 -5 0.90959 host feather1 1 hdd 0.90959 osd.1 -10 2.72876 pod group2 -7 1.81918 host ds1 2 hdd 0.90959

Re: [ceph-users] Where to place Block-DB?

2018-04-26 Thread Konstantin Shalygin
With data located on the OSD (recovery) or as fresh formatted OSD? Thank you. With bluestore NVMe frontend is a part of osd. When frontend dies - backend without db is a junk of bytes. k ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] Running Jewel and Luminous mixed for a longer period

2017-12-30 Thread Konstantin Shalygin
Performance as well - in my testing FileStore was much quicker than BlueStore. Proof? k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] qemu-img convert vs rbd import performance

2017-12-22 Thread Konstantin Shalygin
It's already in qemu 2.9 http://git.qemu.org/?p=qemu.git;a=commit;h=2d9187bc65727d9dd63e2c410b5500add3db0b0d " This patches introduces 2 new cmdline parameters. The -m parameter to specify the number of coroutines running in parallel (defaults to 8). And the -W parameter to allow qemu-img to

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

2017-12-23 Thread Konstantin Shalygin
So finally it logs "scrub ok", but what does " _scan_snaps no head for ..." mean? Does this indicate a problem? Ceph 12.2.2 with bluestore on lvm I think this is because you have snaps created by client before 11.2.1. See http://tracker.ceph.com/issues/19413 I have already come across this

Re: [ceph-users] Bad crc causing osd hang and block all request.

2018-01-08 Thread Konstantin Shalygin
What could cause this problem?Is this caused by a faulty HDD? what data's crc didn't match ? This may be caused due faulty drive. Check your dmesg. ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] Real life EC+RBD experience is required

2018-01-09 Thread Konstantin Shalygin
Hello. My real life experience tells me that this kind of setup will use much more hardware resources and will show lower benchmarks compared to recommended replicated pools on the same hardware. Writes to ec in some cases better than replicated pools.

Re: [ceph-users] replace failed disk in Luminous v12.2.2

2018-01-11 Thread Konstantin Shalygin
Now wonder what is the correct way to replace a failed OSD block disk? Generic way for maintenance (e.g. disk replace) is rebalance by change osd weight: ceph osd crush reweight osdid 0 cluster migrate data "from this osd" When HEALTH_OK you can safe remove this OSD: ceph osd out osd_id

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

2018-01-04 Thread Konstantin Shalygin
On 01/04/2018 11:53 PM, Stefan Kooman wrote: OpenNebula 5.4.3 (issuing rbd commands to ceph cluster). Yes! And what librbd is installed on "commands issuer"? k ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

2018-01-04 Thread Konstantin Shalygin
On 01/04/2018 11:38 PM, Stefan Kooman wrote: Only luminous clients. Mostly rbd (qemu-kvm) images. Who is managed your images? May be OpenStack Cinder? k ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] Suggestion fur naming RBDs

2018-01-16 Thread Konstantin Shalygin
Hi, I was wondering what naming scheme you use for naming RBDs in different pools. There are no strict rules I know, so what might be a best practice? Something like the target service like fileserver_students or webservers_xen, webservers_vmware? A good naming scheme might be helpful :)

Re: [ceph-users] Migration from "classless pre luminous" to"deviceclasses" CRUSH.

2018-02-01 Thread Konstantin Shalygin
On 02/01/2018 08:56 PM, David Turner wrote: You can attempt to mitigate this by creating new, duplicate rules and change 1 pool at a time to start using them. Yes, I'm already prepared to this strategy. k ___ ceph-users mailing list

Re: [ceph-users] Migration from "classless pre luminous" to"deviceclasses" CRUSH.

2018-02-03 Thread Konstantin Shalygin
Migration was complete flawless without any issues and slow requests. Thanks. k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] luminous rbd feature 'striping' is deprecated or just a bug?

2018-01-28 Thread Konstantin Shalygin
Anybody know about changes in rbd feature 'striping'? May be is deprecated feature? What I mean: I have volume created by Jewel client on Luminous cluster. # rbd --user=cinder info solid_rbd/volume-12b5df1e-df4c-4574-859d-22a88415aaf7 rbd image 'volume-12b5df1e-df4c-4574-859d-22a88415aaf7':  

Re: [ceph-users] POOL_NEARFULL

2018-01-29 Thread Konstantin Shalygin
On 01/29/2018 04:25 PM, Karun Josy wrote: In Luminous version, we have to use osd set command Yep. Since Luminous _full options saved in osdmap. k ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] luminous rbd feature 'striping' is deprecated or just a bug?

2018-01-29 Thread Konstantin Shalygin
On 01/29/2018 08:33 PM, Jason Dillaman wrote: OK -- but that is the normal case of RBD w/o the need for fancy striping (i.e. no need for the special feature bit). The striping feature is only needed when using stripe counts != 1 and stripe units != object size. When you specify the

Re: [ceph-users] luminous rbd feature 'striping' is deprecated or just a bug?

2018-01-29 Thread Konstantin Shalygin
On 01/29/2018 07:49 PM, Jason Dillaman wrote: To me, it didn't make sense to set the striping feature bit if fancy striping wasn't really being used. The same logic was applied to the "data-pool" feature bit -- it does make sense to set it if the data pool is really not different from the base

Re: [ceph-users] luminous rbd feature 'striping' is deprecated or just a bug?

2018-01-29 Thread Konstantin Shalygin
On 01/29/2018 06:40 PM, Ilya Dryomov wrote: Unless you specify a non-default stripe_unit/stripe_count, striping feature bit is not set and striping-related fields aren't displayed. This behaviour is new in luminous, but jewel and older clients still work with luminous images. Yes, I see it...

[ceph-users] Migration from "classless pre luminous" to "device classes" CRUSH.

2018-02-01 Thread Konstantin Shalygin
Hi cephers. I have typical double root crush - for nvme pools and hdd pools created on Kraken cluster (what I mean: http://cephnotes.ksperis.com/blog/2015/02/02/crushmap-example-of-a-hierarchical-cluster-map). Now cluster upgraded to Luminous and going to devices classes crush rules and I

Re: [ceph-users] Migration from "classless pre luminous" to"deviceclasses" CRUSH.

2018-02-01 Thread Konstantin Shalygin
We had a MASSIVE data movement upon changing the crush rules to device class based one. I'm not sure about the exact reasons, but I assume that the order of hosts in the crush tree has changed (hosts are ordered lexically now...), which resulted in about 80% of data being moved around. What

Re: [ceph-users] New Ceph-cluster and performance "questions"

2018-02-05 Thread Konstantin Shalygin
/offtopic When and where did you get those? I wonder if they're available again, had 0 luck getting any last year. I was behold P3700 in Russia since December 2017 with real quantity on stock, not just a "price with out of stock".

Re: [ceph-users] max number of pools per cluster

2018-02-11 Thread Konstantin Shalygin
And if for any reason even single PG was damaged and for example stuck inactive - then all RBDs will be affected. First that come to mind is to create a separate pool for every RBD. I think this is insane. Is better to think how Kipod save data in CRUSH. Plan your failure domains and perform

Re: [ceph-users] Ceph-mgr Python error with prometheus plugin

2018-02-16 Thread Konstantin Shalygin
i just try to get the prometheus plugin up and runing Use module from master. From this commit should work with 12.2.2, just wget it and replace stock module. https://github.com/ceph/ceph/blob/d431de74def1b8889ad568ab99436362833d063e/src/pybind/mgr/prometheus/module.py k

Re: [ceph-users] Ceph-mgr Python error with prometheus plugin

2018-02-17 Thread Konstantin Shalygin
On 02/18/2018 09:20 AM, Sergey Malinin wrote: All I got with script replacement is the following: Oooh. Try something earlier, without this function https://github.com/ceph/ceph/blob/1adf4325455ccdbf03da63046c510bb041ac8520/src/pybind/mgr/prometheus/module.py To be clear: prometheus

Re: [ceph-users] Help with Bluestore WAL

2018-02-20 Thread Konstantin Shalygin
Hi, We were recently testing luminous with bluestore. We have 6 node cluster with 12 HDD and 1 SSD each, we used ceph-volume with LVM to create all the OSD and attached with SSD WAL (LVM ). We create individual 10GBx12 LVM on single SDD for each WAL. So all the OSD WAL is on the singe

Re: [ceph-users] Luminous 12.2.3 Changelog ?

2018-02-21 Thread Konstantin Shalygin
Is there any changelog for this release ? https://github.com/ceph/ceph/pull/20503 k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] radosgw + OpenLDAP = Failed the auth strategy, reason=-13

2018-02-19 Thread Konstantin Shalygin
Hi cephers. I try rgw (Luminous 12.2.2) + OpenLDAP. My settings:     "rgw_ldap_binddn": "cn=cisco,ou=people,dc=example,dc=local",     "rgw_ldap_dnattr": "uid",     "rgw_ldap_searchdn": "ou=s3_users,dc=example,dc=local",     "rgw_ldap_searchfilter": "(objectClass=inetOrgPerson)",    

[ceph-users] Any real users of Samsung PM1725 NVMe ?

2018-02-25 Thread Konstantin Shalygin
Hi cephers. I want to know - anybody use PM1725 NVMe drives in production? At this time I plan to upgrade NVMe drives. We learn that is better to avoid "one vendor" or "one model" cluster (http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-December/023463.html). Exception is Intel

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

2018-01-04 Thread Konstantin Shalygin
This is still a pre-production cluster. Most tests have been done using rbd. We did make some rbd clones / snapshots here and there. What clients you used? k ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] radosgw failover help

2018-06-22 Thread Konstantin Shalygin
Has any one, done or working a way to do S3(radosgw) failover. I am trying to work out away to have 2 radosgw servers, with an VIP when one server goes down it will go over to the other. May be better failover + load balancing? For example - nginx do this + TLS. k

Re: [ceph-users] Error: journal specified but not allowed by osd backend

2018-08-02 Thread Konstantin Shalygin
ceph_disk.main.Error: Error: journal specified but not allowed by osd backend I faced this issue once before. The problem is - function is query for osd.0 instead your osd.21. Change in main.py '-i', '0', to 21 (your osd number) '-i', '21', and try again. k

Re: [ceph-users] radosgw: S3 object retention: high usage of default.rgw.log pool

2018-07-30 Thread Konstantin Shalygin
I was misled.In fact, this is not an automatic deletion, but the removal of one object per op by application. Reject. k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Broken multipart uploads

2018-08-05 Thread Konstantin Shalygin
after emtpying the bucket, cannot deleted since there are some aborted multipart uploads radosgw-admin bucket check --bucket=weird_bucket [ "_multipart_DISK_P/collection_1/anonymous/GRLIX/GRLIX_001069.pdf.VOeGNgr-gvhXCrf6dlnhAqhjaFHIF7t.1",

Re: [ceph-users] Broken multipart uploads

2018-08-07 Thread Konstantin Shalygin
On 08/08/2018 01:31 AM, DHD.KOHA wrote: But still, I get No Such key !! s3cmd abortmp s3://weird_bucket 2~CDIJMxZvy8aQejBGBPeNyQK-AJ1lmO4 ERROR: S3 error: 404 (NoSuchKey) s3cmd abortmp s3://weird_bucket 2~alvAZmF5tAlSeiJrUjOwXV7Io22uH0H ERROR: S3 error: 404 (NoSuchKey) Try like this:

Re: [ceph-users] Inconsistent PGs every few days

2018-08-07 Thread Konstantin Shalygin
Hi, I run a cluster with 7 OSD. The cluster has no much traffic on it. But every few days, I get a HEALTH_ERR, because of inconsistent PGs: root at Sam ~ # ceph status cluster: id: c4bfc288-8ba8-4c3a-b3a6-ed95503f50b7

Re: [ceph-users] Help needed for debugging slow_requests

2018-08-15 Thread Konstantin Shalygin
Now here's the thing: Some weeks ago Proxmox upgraded from kernel 4.13 to 4.15. Since then I'm getting slow requests that cause blocked IO inside the VMs that are running on the cluster (but not necessarily on the host with the OSD causing the slow request). If I boot back into 4.13 then Ceph

Re: [ceph-users] Invalid Object map without flags set

2018-08-16 Thread Konstantin Shalygin
We are having issues with ensuring that object-map and fast-diff is working correctly. Most of the time when there is an invalid fast-diff map, the flag is set to correctly indicate this. We have a script that checks for this and rebuilds object maps as required. If we don't fix these,

Re: [ceph-users] Scope of ceph.conf rgw values

2018-08-16 Thread Konstantin Shalygin
I am turning off resharding for Luminous with rgw dynamic resharding = false on the rgw server. When I show the configuration on that server (with ceph daemon), I see that it is false, like I expect. When I show the configuration on the monitor servers, that setting shows up as "true". Do I

Re: [ceph-users] RBD image "lightweight snapshots"

2018-08-13 Thread Konstantin Shalygin
1. Create snapshot of the image we want to backup 2. If there's a previous backup snapshot, export diff and apply it on the backup image 3. If there's no older snapshot, just do a full backup of image So you need incremental backup? Try look to "rbd2qcow2" [1] [1]

Re: [ceph-users] QEMU/Libvirt + librbd issue using Luminous 12.2.7

2018-08-21 Thread Konstantin Shalygin
This issue first started while using Luminous 12.2.5, I upgraded to 12.2.7 and it's still present.  This issue is _not_ present in 12.2.4. With Ceph 12.2.4, using QEMU/KVM + Libvirt, I'm able to mount an rbd image using the following syntax and populated xml: 'virsh attach-device $vm foo.xml

  1   2   3   >