Re: [ceph-users] adding osd node best practice

2015-03-07 Thread Anthony D';Atri
1) That's an awful lot of mons. Are they VM's or something? My sense is that mons >5 have diminishing returns at best. 2) Only two OSD nodes? Assume you aren't running 3 copies of data or racks. 3) The new nodes will have fewer OSD's? Be careful with host / OSD weighting to avoid a gro

Re: [ceph-users] adding osd node best practice

2015-03-12 Thread Anthony D';Atri
> We have application cluster and ceph as storage solution, cluster consists of > six servers, so we've installed > monitor on every one of them, to have ceph cluster sane (quorum) if server or > two of them goes down. You want an odd number for sure, to avoid the classic split-brain problem:

Re: [ceph-users] Sunday's Ceph based business model

2015-03-15 Thread Anthony D';Atri
Interesting idea. I'm not sure, though, Ceph is designed with this sort of latency in mind. Crashplan does let you do something very similar for free, as I understand it, though it's more of a nearline thing. ___ ceph-users mailing list ceph-us

Re: [ceph-users] Scrub Error / How does ceph pg repair work?

2015-05-11 Thread Anthony D';Atri
Agree that 99+% of the inconsistent PG's I see correlate directly to disk flern. Check /var/log/kern.log*, /var/log/messages*, etc. and I'll bet you find errors correlating. -- Anthony ___ ceph-users mailing list ceph-users@lists.ceph.com http://lis

Re: [ceph-users] Scrub Error / How does ceph pg repair work?

2015-05-12 Thread Anthony D';Atri
For me that's true about 1/3 the time, but often I do still have to repair the PG after removing the affected OSD. YMMV. > >> >> >> Agree that 99+% of the inconsistent PG's I see correlate directly to disk >> flern. >> >> Check /var/log/kern.log*, /var/log/messages*, etc. and I'll bet you f

[ceph-users] Survey re journals on SSD vs co-located on spinning rust

2015-01-28 Thread Anthony D';Atri
My apologies if this has been covered ad-naseum in the past; I wasn't finding a lot of relevant archived info. I'm curious how may people are using 1) OSD's on spinning disks, with journals on SSD's -- how many journals per SSD? 4-5? 2) OSD's on spinning disks, with [10GB] journals co-locate

Re: [ceph-users] No auto-mount of OSDs after server reboot

2015-01-30 Thread Anthony D';Atri
One thing than can cause this is messed-up partition ID's / typecodes. Check out the ceph-disk script to see how they get applied. I have a few systems that somehow got messed up -- at boot they don't get started, but if I mounted them manually on /mnt, checked out the whoami file and remoun

Re: [ceph-users] dense storage nodes

2016-05-20 Thread Anthony D';Atri
[ too much to quote ] Dense nodes often work better for object-focused workloads than block-focused, the impact of delayed operations is simply speed vs. a tenant VM crashing. Re RAID5 volumes to decrease the number of OSD’s: This sort of approach is getting increasing attention in that it br

Re: [ceph-users] Do you see a data loss if a SSD hosting several OSD journals crashes

2016-05-20 Thread Anthony D';Atri
> Ceph will not acknowledge a client write before all journals (replica > size, 3 by default) have received the data, so loosing one journal SSD > will NEVER result in an actual data loss. Some say that all replicas must be written; others say that only min_size, 2 by default, must be written be

Re: [ceph-users] Do you see a data loss if a SSD hosting several OSD journals crashes

2016-05-20 Thread Anthony D';Atri
icas must be written or only min_size before > ack? > > But for me the takeaway is that writes are protected - even if the journal > drive crashes, I am covered. > > - epk > > -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On B

Re: [ceph-users] Jewel ubuntu release is half cooked

2016-05-23 Thread Anthony D';Atri
Re: > 2. Inefficient chown documentation - The documentation states that one should > "chown -R ceph:ceph /var/lib/ceph" if one is looking to have ceph-osd ran as > user ceph and not as root. Now, this command would run a chown process one > osd at a time. I am considering my cluster to be a

Re: [ceph-users] Journal partition owner's not change to ceph

2016-06-12 Thread Anthony D';Atri
> The GUID for a CEPH journal partition should be > "45B0969E-9B03-4F30-B4C6-B4B80CEFF106" > I haven't been able to find this info in the documentation on the ceph site The GUID typecodes are listed in the /usr/sbin/ceph-disk script. I had an issue a couple years ago where a subset of OSD’s in o

Re: [ceph-users] Help recovering failed cluster

2016-06-12 Thread Anthony D';Atri
> Current cluster health: >cluster 537a3e12-95d8-48c3-9e82-91abbfdf62e0 > health HEALTH_WARN >5 pgs degraded >8 pgs down >48 pgs incomplete >3 pgs recovering >1 pgs recovery_wait >76 pgs stale >5 pgs stuck d

Re: [ceph-users] Terrible RBD performance with Jewel

2016-07-22 Thread Anthony D';Atri
> FWIW, the xfs -n size=64k option is probably not a good idea. Agreed, moreover it’s a really bad idea. You get memory allocation slowdowns as described in the linked post, and eventually the OSD dies. It can be mitigated somewhat by periodically (say every 2 hours, ymmv) flushing the system

Re: [ceph-users] High iowait on OSD node

2017-07-27 Thread Anthony D';Atri
My first suspicion would be the HBA. Are you using a RAID HBA? If so I suggest checking the status of your BBU/FBWC and cache policy. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Vote re release cadence

2017-09-07 Thread Anthony D';Atri
One vote for: * Drop the odd releases, and aim for a ~9 month cadence. This splits the difference between the current even/odd pattern we've been doing. We've already been bit by gotchas with upgrades even between point releases, so I favor strategies that limit the number of upgrade paths in t

[ceph-users] Anyone else having digest issues with Apple Mail?

2017-09-13 Thread Anthony D';Atri
For a couple of weeks now digests have been appearing to me off and on with a few sets of MIME headers and maybe 1-2 messages. When I look at the raw text the whole digest is in there. Screencap below. Anyone else experiencing this? https://www.evernote.com/l/AL2CMToOPiBIJYZgw9KzswqiBhHHoRIm

Re: [ceph-users] Ceph OSD on Hardware RAID

2017-09-29 Thread Anthony D';Atri
In addition to the points that others made so well: - When using parity RAID, eg. RAID5 to create OSD devices, one reduces aggregate write speed — specially if using HDD’s — due to write amplification. - If using parity or replicated RAID, one might semi-reasonably get away with reducing Ceph’s

Re: [ceph-users] osd create returns duplicate ID's

2017-09-29 Thread Anthony D';Atri
Luis: As others have mentioned, be sure that when you delete an OSD each step is completed successfully: - OSD process is killed - OSD is marked out/down in the CRUSH map - ceph osd crush delete osd.xxx - ceph osd rm osd.xxx - ceph auth del osd.xxx - Also be sure to unmount the /var/lib/ceph mou

Re: [ceph-users] HW Raid vs. Multiple OSD

2017-11-13 Thread Anthony D';Atri
Oscar, a few thoughts: o I think you might have some misunderstandings about how Ceph works. Ceph is best deployed as a single cluster spanning multiple servers, generally at least 3. Is that your plan? It sort of sounds as though you're thinking of Ceph managing only the drives local to eac

Re: [ceph-users] Sizing your MON storage with a large cluster

2018-02-09 Thread Anthony D';Atri
Thanks, Wido -- words to live by. I had all kinds of problems with mon DBs not compacting under Firefly, really pointed out the benefit of having ample space on the mons -- and the necessity of having those DB's live on something faster than an LFF HDD. I've had this happen when using ceph-gent

Re: [ceph-users] Upgrade from Hammer to Jewel

2016-10-25 Thread Anthony D';Atri
> We have an Openstack which use Ceph for Cinder and Glance. Ceph is in > Hammer release and we need to upgrade to Jewel. My question is : > are the Hammer clients compatible with the Jewel servers ? (upgrade of Mon > then Ceph servers first) > As the upgrade of the Ceph client need a reboot of al

Re: [ceph-users] Monitoring Overhead

2016-10-26 Thread Anthony D';Atri
> Collectd and graphite look really nice. Also look into Grafana, and of course RHSC. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Intel P3700 SSD for journals

2016-11-20 Thread Anthony D';Atri
The SATA S3700 series has been the de-facto for journals for some time. And journals don’t neeed all that much space. We’re using 400GB P3700’s. I’ll say a couple of things: o Update to the latest firmware available when you get your drives, qual it and stick with it for a while so you have a

Re: [ceph-users] Intel P3700 SSD for journals

2016-11-22 Thread Anthony D';Atri
ERC 730 raid controller? > > Is there anything particular required when installing the CentOS on > these drives, or they will be automatically detected and work out of > the box by default? Thx will > > On Mon, Nov 21, 2016 at 12:16 PM, Anthony D'Atri wrote: >> The SATA

Re: [ceph-users] cephfs ata1.00: status: { DRDY }

2017-01-06 Thread Anthony D';Atri
YMMV of course, but the first thing that struck me was the constraint of scrub times. Constraining them to fewer hours can mean that more run in parallel. If you truly have off-hours for client ops (Graphite / Grafana are great for visualizing that) that might make sense, but in my 24x7 OpenSt

[ceph-users] Exact scope of OSD heartbeating?

2018-07-17 Thread Anthony D';Atri
The documentation here: http://docs.ceph.com/docs/master/rados/configuration/mon-osd-interaction/ says "Each Ceph OSD Daemon checks the heartbeat of other Ceph OSD Daemons every 6 seconds" and " If a neighboring Ceph

Re: [ceph-users] Exact scope of OSD heartbeating?

2018-07-18 Thread Anthony D';Atri
Thanks, Dan. I thought so but wanted to verify. I'll see if I can work up a doc PR to clarify this. >> The documentation here: >> >> http://docs.ceph.com/docs/master/rados/configuration/mon-osd-interaction/ >> >> says >> >> "Each Ceph OSD Daemon checks the heartbeat of other Ceph OSD Daemons

Re: [ceph-users] Adding node efficient data move.

2018-09-11 Thread Anthony D';Atri
> When adding a node and I increment the crush weight like this. I have > the most efficient data transfer to the 4th node? > > sudo -u ceph ceph osd crush reweight osd.23 1 > sudo -u ceph ceph osd crush reweight osd.24 1 > sudo -u ceph ceph osd crush reweight osd.25 1 > sudo -u ceph ceph osd

Re: [ceph-users] Favorite SSD

2018-09-17 Thread Anthony D';Atri
> Micron 5200 line seems to not have a high endurance SKU like the 5100 line > sadly. The 3.84TB 5200 PRO is rated at ~2.4 DWPD — you need higher than that? I do find references to higher-durability ~5DWPD 5200 MAX models up to 1.9 TB. Online resources on the 5200 product line don’t always a

[ceph-users] warning: fast-diff map is invalid operation may be slow; object map invalid

2018-10-15 Thread Anthony D';Atri
We turned on all the RBD v2 features while running Jewel; since then all clusters have been updated to Luminous 12.2.2 and additional clusters added that have never run Jewel. Today I find that a few percent of volumes in each cluster have issues, examples below. I'm concerned that these issu

Re: [ceph-users] Migrating to new pools

2018-02-21 Thread Anthony D';Atri
>> I was thinking we might be able to configure/hack rbd mirroring to mirror to >> a pool on the same cluster but I gather from the OP and your post that this >> is not really possible? > > No, it's not really possible currently and we have no plans to add > such support since it would not be of a

Re: [ceph-users] Migrating to new pools

2018-02-22 Thread Anthony D';Atri
Cumulative followup to various insightful replies. I wrote: No, it's not really possible currently and we have no plans to add >>> such support since it would not be of any long-term value. >> >> The long-term value would be the ability to migrate volumes from, say, a >> replicated pool to

Re: [ceph-users] IO rate-limiting with Ceph RBD (and libvirt)

2018-03-22 Thread Anthony D';Atri
> FYI: I/O limiting in combination with OpenStack 10/12 + Ceph doesn?t work > properly. Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1476830 > That's an OpenStack bug, nothing to do with Ceph. Nothing stops you from using virsh to throt

Re: [ceph-users] ceph-deploy: recommended?

2018-04-06 Thread Anthony D';Atri
> ?I read a couple of versions ago that ceph-deploy was not recommended > for production clusters.? InkTank had sort of discouraged the use of ceph-deploy; in 2014 we used it only to deploy OSDs. Some time later the message changed. ___ ceph-users m

Re: [ceph-users] mon.mon01 store is getting too big! 18119 MB >= 15360 MB -- 94% avail

2017-02-01 Thread Anthony D';Atri
> In particular, when using leveldb, stalls while reading or writing to > the store - typically, leveldb is compacting when this happens. This > leads to all sorts of timeouts to be triggered, but the really annoying > one would be the lease timeout, which tends to result in flapping quorum. >

[ceph-users] http://www.dell.com/support/home/us/en/04/product-support/servicetag/JFGQY02/warranty#

2017-03-10 Thread Anthony D';Atri
> As long as you don?t nuke the OSDs or the journals, you should be OK. This. Most HBA failures I’ve experienced don’t corrupt data on the drives, bit it can happen. Assuming the data is okay, you should be able to just install the OS, install the *same version* of Ceph packages, reboot, and h

[ceph-users] apologies for the erroneous subject - should have been Re: Unable to boot OS on cluster node

2017-03-11 Thread Anthony D';Atri
A certain someone bumped my elbow as I typed, think in terms of this week’s family-bombed video going the rounds on FB. My ignominy is boundless and my door now locked when replying. — aad ___ ceph-users mailing list ceph-users@lists.ceph.com http

Re: [ceph-users] Directly addressing files on individual OSD

2017-03-15 Thread Anthony D';Atri
As I parse Youssef’s message, I believe there are some misconceptions. It might help if you could give a bit more info on what your existing ‘cluster’ is running. NFS? CIFS/SMB? Something else? 1) Ceph regularly runs scrubs to ensure that all copies of data are consistent. The checksumming

Re: [ceph-users] What's the actual justification for min_size?

2017-03-21 Thread Anthony D';Atri
>> a min_size of 1 is dangerous though because it means you are 1 hard disk >> failure away from losing the objects within that placement group entirely. a >> min_size of 2 is generally considered the minimum you want but many people >> ignore that advice, some wish they hadn't. > > I admit I

Re: [ceph-users] add multiple OSDs to cluster

2017-03-21 Thread Anthony D';Atri
Deploying or removing OSD’s in parallel for sure can save elapsed time and avoid moving data more than once. There are certain pitfalls, though, and the strategy needs careful planning. - Deploying a new OSD at full weight means a lot of write operations. Running multiple whole-OSD backfills

Re: [ceph-users] What's the actual justification for min_size?

2017-03-21 Thread Anthony D';Atri
ses now. :) > So: just don't ever set min_size to 1. > -Greg > On Tue, Mar 21, 2017 at 6:04 PM Anthony D'Atri wrote: > >> a min_size of 1 is dangerous though because it means you are 1 hard disk > >> failure away from losing the objects within that placement gr

Re: [ceph-users] SSD Primary Affinity

2017-04-18 Thread Anthony D';Atri
I get digests, so please forgive me if this has been covered already. > Assuming production level, we would keep a pretty close 1:2 SSD:HDD ratio, 1:4-5 is common but depends on your needs and the devices in question, ie. assuming LFF drives and that you aren’t using crummy journals. > First of

Re: [ceph-users] SSD Primary Affinity

2017-04-19 Thread Anthony D';Atri
Re ratio, I think you’re right. Write performance depends for sure on what the journal devices are. If the journals are colo’d on spinners, then for sure the affinity game isn’t going to help writes massively. My understanding of write latency is that min_size journals have to be written befo

Re: [ceph-users] Sharing SSD journals and SSD drive choice

2017-04-26 Thread Anthony D';Atri
At a meeting with Intel folks a while back, they discussed the idea that future large devices — which we’re starting to now see — would achieve greater *effective* durability via a lower cost/GB that encourages the use of larger than needed devices. Which is a sort of overprovisioning, just mor

Re: [ceph-users] Re-weight Entire Cluster?

2017-05-30 Thread Anthony D';Atri
> It appears the current best practice is to weight each OSD according to it?s > size (3.64 for 4TB drive, 7.45 for 8TB drive, etc). OSD’s are created with those sorts of CRUSH weights by default, yes. Which is convenient, but it’s import to know that those weights are arbitrary, and what re

Re: [ceph-users] Re-weight Entire Cluster?

2017-05-30 Thread Anthony D';Atri
up 1.0 1.0 > 221 1.0 osd.221 up 1.0 1.0 > 222 1.0 osd.222 up 1.0 1.0 > 223 1.0 osd.223 up 1.0 1.0 > > -Original Message--

Re: [ceph-users] Crushmap from Rack aware to Node aware

2017-06-02 Thread Anthony D';Atri
All very true and worth considering, but I feel compelled to mention the strategy of setting mon_osd_down_out_subtree_limit carefully to prevent automatic rebalancing. *If* the loss of a failure domain is temporary, ie. something you can fix fairly quickly, it can be preferable to not start tha

Re: [ceph-users] SSD Journal

2016-01-29 Thread Anthony D';Atri
> Right now we run the journal as a partition on the data disk. I've build > drives without journals and the write performance seems okay but random io > performance is poor in comparison to what it should be. Co-located journals have multiple issues: o The disks are presented with double the

Re: [ceph-users] Bluestore nvme DB/WAL size

2018-12-21 Thread Anthony D';Atri
> It'll cause problems if yours the only one NVMe drive will die - you'll lost > all the DB partitions and all the OSDs are going to be failed The severity of this depends a lot on the size of the cluster. If there are only, say, 4 nodes total, for sure the loss of a quarter of the OSDs will b

[ceph-users] Questions re mon_osd_cache_size increase

2019-01-07 Thread Anthony D';Atri
> * The default memory utilization for the mons has been increased > somewhat. Rocksdb now uses 512 MB of RAM by default, which should > be sufficient for small to medium-sized clusters; large clusters > should tune this up. Also, the `mon_osd_cache_size` has been > increase from 10 OSDMap

Re: [ceph-users] Questions re mon_osd_cache_size increase

2019-01-07 Thread Anthony D';Atri
Thanks, Greg. This is as I suspected. Ceph is full of subtleties and I wanted to be sure. -- aad > > The osd_map_cache_size controls the OSD’s cache of maps; the change in 13.2.3 > is to the default for the monitors’. > On Mon, Jan 7, 2019 at 8:24 AM Anthony D

Re: [ceph-users] Ceph Nautilus Release T-shirt Design

2019-01-17 Thread Anthony D';Atri
>> Lenz has provided this image that is currently being used for the 404 >> page of the dashboard: >> >> https://github.com/ceph/ceph/blob/master/src/pybind/mgr/dashboard/frontend/src/assets/1280px-Nautilus_Octopus.jpg > > Nautilus *shells* are somewhat iconic/well known/distinctive. Maybe a > v

Re: [ceph-users] Questions about using existing HW for PoC cluster

2019-01-27 Thread Anthony D';Atri
> Been reading "Learning Ceph - Second Edition” An outstanding book, I must say ;) > So can I get by with using a single SATA SSD (size?) per server for RocksDB / > WAL if I'm using Bluestore? Depends on the rest of your setup and use-case, but I think this would be a bottleneck. Some thought

Re: [ceph-users] ceph mon_data_size_warn limits for large cluster

2019-02-18 Thread Anthony D';Atri
On older releases, at least, inflated DBs correlated with miserable recovery performance and lots of slow requests. The DB and OSDs were also on HDD FWIW. A single drive failure would result in substantial RBD impact. > On Feb 18, 2019, at 3:28 AM, Dan van der Ster wrote: > > Not really.

Re: [ceph-users] Ceph cluster stability

2019-02-22 Thread Anthony D';Atri
? Did we start recommending that production mons run on a VM? I'd be very hesitant to do that, though probably some folks do. I can say for sure that in the past (Firefly) I experienced outages related to mons running on HDDs. That was a cluster of 450 HDD OSDs with colo journals and hundred

Re: [ceph-users] [Bluestore] Some of my osd's uses BlueFS slow storage for db - why?

2019-02-24 Thread Anthony D';Atri
> Date: Fri, 22 Feb 2019 16:26:34 -0800 > From: solarflow99 > > > Aren't you undersized at only 30GB? I thought you should have 4% of your > OSDs The 4% guidance is new. Until relatively recently the oft-suggested and default value was 1%. > From: "Vitaliy Filippov" > Numbers are easy to

Re: [ceph-users] Ceph block storage cluster limitations

2019-03-30 Thread Anthony D';Atri
> Hello, > > I wanted to know if there are any max limitations on > > - Max number of Ceph data nodes > - Max number of OSDs per data node > - Global max on number of OSDs > - Any limitations on the size of each drive managed by OSD? > - Any limitation on number of client nodes? > - Any limitatio

Re: [ceph-users] rgw, nss: dropping the legacy PKI token support in RadosGW (removed in OpenStack Ocata)

2019-04-19 Thread Anthony D';Atri
I've been away from OpenStack for a couple of years now, so this may have changed. But back around the Icehouse release, at least, upgrading between OpenStack releases was a major undertaking, so backing an older OpenStack with newer Ceph seems like it might be more common than one might thin

Re: [ceph-users] clock skew

2019-04-26 Thread Anthony D';Atri
> @Janne: i will checkout/implement the peer config per your suggestion. > However what confuses us is that chrony thinks the clocks match, and > only ceph feels it doesn't. So we are not sure if the peer config will > actually help in this situation. But time will tell. Ar ar. Chrony thinks t

Re: [ceph-users] Observation of bluestore db/wal performance

2019-07-21 Thread Anthony D';Atri
This may be somewhat controversial, so I’ll try to tread lightly. Might we infer that your OSDs are on spinners? And at 500 GB it would seem likely that they and the servers are old? Please share hardware details and OS. Having suffered an “enterprise” dogfood deployment in which I had to atte

Re: [ceph-users] New best practices for osds???

2019-07-25 Thread Anthony D';Atri
> We run few hundred HDD OSDs for our backup cluster, we set one RAID 0 per HDD > in order to be able > to use -battery protected- write cache from the RAID controller. It really > improves performance, for both > bluestore and filestore OSDs. Having run something like 6000 HDD-based FileStore O

Re: [ceph-users] New best practices for osds???

2019-07-26 Thread Anthony D';Atri
> This is worse than I feared, but very much in the realm of concerns I > had with using single-disk RAID0 setups.? Thank you very much for > posting your experience!? My money would still be on using *high write > endurance* NVMes for DB/WAL and whatever I could afford for block.? yw. Of cou

Re: [ceph-users] How to add 100 new OSDs...

2019-07-28 Thread Anthony D';Atri
Paul Emmerich wrote: > +1 on adding them all at the same time. > > All these methods that gradually increase the weight aren't really > necessary in newer releases of Ceph. Because the default backfill/recovery values are lower than they were in, say, Dumpling? Doubling (or more) the size of a

Re: [ceph-users] How to add 100 new OSDs...

2019-08-04 Thread Anthony D';Atri
>>> We have been using: >>> >>> osd op queue = wpq >>> osd op queue cut off = high >>> >>> It virtually eliminates the impact of backfills on our clusters. Our > > It does better because it is a fair share queue and doesn't let recovery > ops take priority over client ops at any point for any ti

Re: [ceph-users] How to maximize the OSD effective queue depth in Ceph?

2019-08-06 Thread Anthony D';Atri
> However, I'm starting to think that the problem isn't with the number > of threads that have work to do... the problem may just be that the > OSD & PG code has enough thread locking happening that there is no > possible way to have more than a few things happening on a single OSD > (or perhaps a

Re: [ceph-users] WAL/DB size

2019-08-14 Thread Anthony D';Atri
Good points in both posts, but I think there’s still some unclarity. Absolutely let’s talk about DB and WAL together. By “bluestore goes on flash” I assume you mean WAL+DB? “Simply allocate DB and WAL will appear there automatically” Forgive me please if this is obvious, but I’d like to see a

Re: [ceph-users] WAL/DB size

2019-08-16 Thread Anthony D';Atri
udes WAL, see >> http://yourcmc.ru/wiki/Ceph_performance#About_block.db_sizing >> >> 15 августа 2019 г. 1:15:58 GMT+03:00, Anthony D'Atri >> пишет: >>> >>> Good points in both posts, but I think there’s still some unclarity. ___

Re: [ceph-users] PG Balancer Upmap mode not working

2019-12-09 Thread Anthony D';Atri
> How is that possible? I dont know how much more proof I need to present that > there's a bug. FWIW, your pastes are hard to read with all the ? in them. Pasting non-7-bit-ASCII? > |I increased PGs and see no difference. From what pgp_num to what new value? Numbers that are not a power of 2

Re: [ceph-users] ceph-users Digest, Vol 83, Issue 18

2019-12-20 Thread Anthony D';Atri
> Hi Sinan, > > I would not recommend using 860 EVO or Crucial MX500 SSD's in a Ceph cluster, > as those are consumer grade solutions and not enterprise ones. The OP knows that, but wants to know why. > Performance and durability will be issues. If feasible, I would simply go > NVMe as it sou

Re: [ceph-users] Consumer-grade SSD in Ceph

2020-01-03 Thread Anthony D';Atri
>> SATA: Micron 5100-5200-5300, Seagate Nytro 1351/1551 (don't forget to >> disable their cache with hdparm -W 0) We didn’t find a measurable difference doing this on 5100s, ymmv. Depending on your use-case, CRUSH rules (EC vs R), etc. sub-DWPD models may be fine for OSDs, but I suggest higher