Re: [ceph-users] Cluster stuck in failed state after power failure - please help

how". So problem is different, > that my mgr or mds is not running. > > With regards > Jan Pekar > > On 11.12.2017 19:42, David Turner wrote: > > It honestly just looks like your MDS and MGR daemons are not configured > > to start automatically. Try starting them manua

Re: [ceph-users] Cluster stuck in failed state after power failure - please help

It honestly just looks like your MDS and MGR daemons are not configured to start automatically. Try starting them manually and then if that fixes the things, go through and enable them to start automatically. Assuming you use systemctl the commands to check and fix this would be something like

Re: [ceph-users] public/cluster network

Hi Roman Whilst you can define multiple subnets in the public network directive, the MONs still only bind to a single IP. Your clients need to be able to route to that IP. From what you're saying, 172.x.x.x/24 is an isolated network, so a client on the 10.x.x.x network is not going to be able to

Re: [ceph-users] questions about rbd image

An RBD can only be mapped to a single client host. There is no way around this. An RBD at its core is a block device. Connecting an RBD to 2 servers would be like connecting a harddrive to 2 servers. On Mon, Dec 11, 2017 at 9:13 AM 13605702596 <13605702...@163.com> wrote: > hi Jason > thanks

Re: [ceph-users] The way to minimize osd memory usage?

The docs recommend 1GB/TB of OSDs. I saw people asking if this was still accurate for bluestore and the answer was that it is more true for bluestore than filestore. There might be a way to get this working at the cost of performance. I would look at Linux kernel memory settings as much as ceph

Re: [ceph-users] Removing a ceph node and ceph documentation.

Master is the current development version available from git. You have to compile it yourself to use that version. It's what the developers are currently working on, but hasn't been merged into a release yet. On Sat, Dec 9, 2017, 12:33 PM Sameer S <mailboxtosam...@gmail.com> wrote: >

Re: [ceph-users] Removing a ceph node and ceph documentation.

To get specific versions of the ceph documentation you change the version in the URL to yours. If you see giant, kraken, matter, Luminous, Hammer, Jewel, etc in the URL right after docs then change it to the one you want. The following URL is for master. You would replace the master after docs

Re: [ceph-users] upgrade from kraken 11.2.0 to 12.2.2 bluestore EC

Why are you rebooting the node? You should only need to restart the ceph services. You need all of your MONs to be running Luminous before any Luminous OSDs will be accepted by the cluster. So you should update the packages on each server, restart the MONs, then restart your OSDs. After you

Re: [ceph-users] cephfs monitor I/O and throughput

We have graphs for network usage in graphana. We even have aggregate graphs for projects. For my team, we specifically have graphs for the Ceph cluster osd public network, osd private network, rgw network, and mon network. You should be able to do something similar for each of the servers in

Re: [ceph-users] How to remove a faulty bucket?

> The garbage collection is already listed as empty. > > Regards, > > Martin > > Von: David Turner <drakonst...@gmail.com> > Datum: Freitag, 8. Dezember 2017 um 15:19 > An: Martin Emrich <martin.emr...@empolis.com> > Cc: ceph-users <ceph-users@lists.ceph.com> >

Re: [ceph-users] How to remove a faulty bucket? [WAS:Re: Resharding issues / How long does it take?]

First off, you can rename a bucket and create a new one for the application to use. You can also unlink the bucket so it is no longer owned by the access-key/user that created it. That should get your application back on its feet. I have had very little success with bypass-gc, although I think it

Re: [ceph-users] rbd-nbd timeout and crash

Do you have the FS mounted with a trimming ability? What are your mount options? On Wed, Dec 6, 2017 at 5:30 PM Jan Pekař - Imatic wrote: > Hi, > > On 6.12.2017 15:24, Jason Dillaman wrote: > > On Wed, Dec 6, 2017 at 3:46 AM, Jan Pekař - Imatic >

Re: [ceph-users] Sudden omap growth on some OSDs

I have no proof or anything other than a hunch, but OSDs don't trim omaps unless all PGs are healthy. If this PG is actually not healthy, but the cluster doesn't realize it while these 11 involved OSDs do realize that the PG is unhealthy... You would see this exact problem. The OSDs think a PG

Re: [ceph-users] I cannot make the OSD to work, Journal always breaks 100% time

Why are you flushing the journal after you zero it instead of before? That does nothing. You want to flush the journal while it has objects that might not be on the osd before you zero it. On Wed, Dec 6, 2017, 6:02 AM Ronny Aasen wrote: > On 06. des. 2017 10:01,

Re: [ceph-users] List directory in cephfs blocking very long time

The 3.10 kernel is very old compared to 12.2.2. I would recommend trying a newer kernel or using ceph-fuse. I personally use ceph-fuse. It is updated with each release of Ceph and will match the new features released more closely than the kernel driver. On Tue, Dec 5, 2017, 6:59 AM 张建

Re: [ceph-users] List directory in cephfs blocking very long time

Not seen this myself but you should update to at least CentOS 7.3, ideally 7.4. I believe a lot of cephfs fixes went into those kernels. If you still have the issue with the CentOS kernels, test with the latest upstream kernel. And/or test with latest Fuse client. On Tue, Dec 5, 2017 at 12:01 PM,

Re: [ceph-users] Adding multiple OSD

> m=3 > plugin=jerasure > technique=reed_sol_van > w=8 > > > Cluster has 8 nodes, with 3 disks each. We are planning to add 2 more on > each nodes. > > If I understand correctly, then I can add 3 disks at once right , assuming > 3 disks can fail at a time as per the ec co

Re: [ceph-users] Adding multiple OSD

Depending on how well you burn-in/test your new disks, I like to only add 1 failure domain of disks at a time in case you have bad disks that you're adding. If you are confident that your disks aren't likely to fail during the backfilling, then you can go with more. I just added 8 servers (16

Re: [ceph-users] HELP with some basics please

"The journals can only be moved back by a complete rebuild of that osd as to my knowledge." I'm assuming that since this is a cluster that he's inherited and that it's configured like this that it's probably not running luminous or bluestore OSDs. Again more information needed about your cluster

Re: [ceph-users] HELP with some basics please

ce there is > NO read error on any bit on the living host) I could (theoretically) > recover, is that right? > OR is it that if any two disks in the cluster fail at the same time > (or while one is still being rebuild) all my data would be gone? > > > > On Mon, Dec 4,

Re: [ceph-users] Replaced a disk, first time. Quick question

On Mon, Dec 4, 2017 at 4:39 PM, Drew Weaver wrote: > Howdy, > > > > I replaced a disk today because it was marked as Predicted failure. These > were the steps I took > > > > ceph osd out osd17 > > ceph -w #waited for it to get done > > systemctl stop ceph-osd@osd17 > >

Re: [ceph-users] dropping trusty

On 12/04/2017 01:12 AM, kefu chai wrote: > On Fri, Dec 1, 2017 at 1:55 AM, David Galloway <dgall...@redhat.com> wrote: >> On 11/30/2017 12:21 PM, Sage Weil wrote: >>> We're talking about dropping trusty support for mimic due to the old >>> compiler (incomplete C

Re: [ceph-users] HELP with some basics please

Your current node configuration cannot do size=3 for any pools. You only have 2 hosts with HDDs and 2 hosts with SSDs in each root. You cannot put 3 copies of data for an HDD pool on 3 separate nodes when you only have 2 nodes with HDDs... In this configuration, size=2 is putting a copy of the

Re: [ceph-users] Ceph+RBD+ISCSI = ESXI issue

rt-task requests are being sent by the initiator after tripping a local I/O timeout. Your vmkernel logs should shed a bit more light on this. Cheers, David ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] dropping trusty

On 11/30/2017 12:21 PM, Sage Weil wrote: > We're talking about dropping trusty support for mimic due to the old > compiler (incomplete C++11), hassle of using an updated toolchain, general > desire to stop supporting old stuff, and lack of user objections to > dropping it in the next release. >

Re: [ceph-users] rbd mount unmap network outage

This doesn't answer your question, but maybe nudges you in a different direction. CephFS seams like the much better solution for what you're doing. You linked a 5 year old blog post. CephFS was not a stable technology at the time, but it's an excellent method to share a network FS to multiple

Re: [ceph-users] CRUSH rule seems to work fine not for all PGs in erasure coded pools

active+clean+remapped is not a healthy state for a PG. If it actually we're going to a new osd it would say backfill+wait or backfilling and eventually would get back to active+clean. I'm not certain what the active+clean+remapped state means. Perhaps a PG query, PG dump, etc can give more

Re: [ceph-users] "failed to open ino"

On Tue, Nov 28, 2017 at 1:50 PM, Jens-U. Mozdzen <jmozd...@nde.ag> wrote: > Hi David, > > Zitat von David C <dcsysengin...@gmail.com>: > >> On 27 Nov 2017 1:06 p.m., "Jens-U. Mozdzen" <jmozd...@nde.ag> wrote: >> >> Hi David, >&g

Re: [ceph-users] force scrubbing

I personally set max_scrubs to 0 on the cluster and then set it to 1 only on the osds involved in the PG you want to scrub. Setting the cluster to max_scrubs of 1 and then upping the involved osds to 2 might help, but is not a guarantee. On Tue, Nov 28, 2017 at 7:25 PM Gregory Farnum

Re: [ceph-users] ceph-disk is now deprecated

Isn't marking something as deprecated meaning that there is a better option that we want you to use and you should switch to it sooner than later? I don't understand how this is ready to be marked as such if ceph-volume can't be switched to for all supported use cases. If ZFS, encryption, FreeBSD,

Re: [ceph-users] CRUSH rule seems to work fine not for all PGs in erasure coded pools

Your EC profile requires 5 servers to be healthy. When you remove 1 OSD from the cluster, it recovers by moving all of the copies on that OSD to other OSDs in the same host. However when you remove an entire host, it cannot store 5 copies of the data on the 4 remaining servers with your crush

Re: [ceph-users] install ceph-osd failed in docker

3:10:09AM +0000, David Turner wrote: > > Disclaimer... This is slightly off topic and a genuine question. I am a > > container noobie that has only used them for test environments for nginx > > configs and ceph client multi-tenency benchmarking. > > > > I understand t

Re: [ceph-users] ceph all-nvme mysql performance tuning

all NVMe, environment. David Byte Sr. Technology Strategist SCE Enterprise Linux SCE Enterprise Storage Alliances and SUSE Embedded db...@suse.com 918.528.4422 From: ceph-users <ceph-users-boun...@lists.ceph.com> on behalf of German Anders <gand...@despegar.com> Date: Monday, November 2

Re: [ceph-users] "failed to open ino"

On 27 Nov 2017 1:06 p.m., "Jens-U. Mozdzen" <jmozd...@nde.ag> wrote: Hi David, Zitat von David C <dcsysengin...@gmail.com>: Hi Jens > > We also see these messages quite frequently, mainly the "replicating > dir...". Only seen "failed

Re: [ceph-users] "failed to open ino"

Hi Jens We also see these messages quite frequently, mainly the "replicating dir...". Only seen "failed to open ino" a few times so didn't do any real investigation. Our set up is very similar to yours, 12.2.1, active/standby MDS and exporting cephfs through KNFS (hoping to replace with Ganesha

Re: [ceph-users] CephFS 12.2.0 -> 12.2.1 change in inode caching behaviour

Yep, that did it! Thanks, Zheng. I should read release notes more carefully! On Fri, Nov 24, 2017 at 7:09 AM, Yan, Zheng <uker...@gmail.com> wrote: > On Thu, Nov 23, 2017 at 9:17 PM, David C <dcsysengin...@gmail.com> wrote: > > Hi All > > > > I upgraded my 12.2.0

Re: [ceph-users] install ceph-osd failed in docker

Disclaimer... This is slightly off topic and a genuine question. I am a container noobie that has only used them for test environments for nginx configs and ceph client multi-tenency benchmarking. I understand the benefits to containerizing RGW, MDS, and MGR daemons. I can even come up with a

Re: [ceph-users] ceph osd after xfs repair only 50 percent data and osd won't start

If you are too a point where you need to repair the xfs partition, you should probably just rebuild the osd and backfill back onto it as a fresh osd. That's even more true now that the repair had bad side effects. On Sat, Nov 25, 2017, 11:33 AM Hauke Homburg wrote: >

Re: [ceph-users] Admin server

An admin node does not have any bearing on the running of the cluster. Usually they're helpful for centralized monitoring, deploying, and management... But none of that involves a service needed by the cluster or information any daemon in the cluster needs. On Thu, Nov 23, 2017, 1:08 PM Karun

Re: [ceph-users] Journal / WAL drive size?

provision 2GB and haven’t experienced any issues with that. You also probably will need to adjust the ratios, but that was covered in other threads previously. David Byte Sr. Technical Strategist IHV Alliances and Embedded SUSE Sent from my iPhone. Typos are Apple's fault. On Nov 23, 2017, at 3:19 PM

[ceph-users] CephFS 12.2.0 -> 12.2.1 change in inode caching behaviour

280168, > > "traverse_hit": 185739606, > > "traverse_forward": 0, > > "traverse_discover": 0, > > "traverse_dir_fetch": 118150, > > "traverse_remote_ino": 8, > > "traverse_lock": 60

Re: [ceph-users] two keys for one single uid

ample? > > Best Regards, > > 2017-11-23 9:55 GMT-02:00 Abhishek <abhis...@suse.com>: > >> On 2017-11-23 12:41, Daniel Picolli Biazus wrote: >> >>> Hey David, >>> >> >> You can create multiple keys using key create command >>

Re: [ceph-users] two keys for one single uid

If you create a subuser of the uid, then the subuser can have its own name and key while being the same user. You can also limit a subuser to read, write, read+write, or full permissions. Full is identical permissions for the subuser as the user. Full enables creating and deleting buckets. To

Re: [ceph-users] OSD is near full and slow in accessing storage from client

Yes, increasing the PG count for the data pool will be what you want to do when you add osds to your cluster. On Wed, Nov 22, 2017, 9:25 AM gjprabu <gjpr...@zohocorp.com> wrote: > Hi David, > > Thanks, will check osd weight settings and we are not using rbd > and w

Re: [ceph-users] radosgw bucket rename and change owner

User and bucket operations have more to do with what is providing the S3 API. In this case you're using swift for that. The Ceph tools to do this would be if you're using RGW to provide the S3 API. The answers you're looking for would be in how to do this with SWIFT, if I'm not mistaken. Ceph

Re: [ceph-users] HEALTH_ERR pgs are stuck inactive for more than 300 seconds

All you have to do is figure out why osd.0, osd.1, and osd.2 are down and get the daemons running. They have PGs assigned to them, but since they are not up and running those PGs are in a down state. You can check the logs for them in /var/log/ceph/. Did you have any errors when deploying these

Re: [ceph-users] OSD is near full and slow in accessing storage from client

and/or reweight of the osd to help the algorithm balance that out. On Tue, Nov 21, 2017 at 12:11 AM gjprabu <gjpr...@zohocorp.com> wrote: > Hi David, > >This is our current status. > > > ~]# ceph status > cluster b466e09c-f7ae-4e89-99a7-99d30eba0a13 > health

Re: [ceph-users] OSD is near full and slow in accessing storage from client

What is your current `ceph status` and `ceph df`? The status of your cluster has likely changed a bit in the last week. On Mon, Nov 20, 2017 at 6:00 AM gjprabu <gjpr...@zohocorp.com> wrote: > Hi David, > > Sorry for the late reply and its completed OSD Sync and mo

Re: [ceph-users] Deleting large pools

I created a bug tracker for this here. http://tracker.ceph.com/issues/22201 Thank you for your help Gregory. On Sat, Nov 18, 2017 at 9:20 PM Gregory Farnum <gfar...@redhat.com> wrote: > On Wed, Nov 15, 2017 at 6:50 AM David Turner <drakonst...@gmail.com> > wrote: > >&g

Re: [ceph-users] Ceph - SSD cluster

This topic has been discussed in detail multiple times and from various angles. Your key points are going to be CPU limits iops, dwpd, iops vs bandwidth, and SSD clusters/pools in general. You should be able to find everything you need in the archives. On Mon, Nov 20, 2017, 12:56 AM M Ranga Swami

Re: [ceph-users] OSD Random Failures - Latest Luminous

()+0x7494) [0x7fb45cab4494] > > 17: (clone()+0x3f) [0x7fb45bb3baff] > > NOTE: a copy of the executable, or `objdump -rdS ` is needed > to interpret this. > > > > I guess even with noup the OSD/PG still has the peer with the other PG’s > which is the stage that causes t

Re: [ceph-users] OSD Random Failures - Latest Luminous

Does letting the cluster run with noup for a while until all down disks are idle, and then letting them come in help at all? I don't know your specific issue and haven't touched bluestore yet, but that is generally sound advice when is won't start. Also is there any pattern to the osds that are

Re: [ceph-users] Restart is required?

The filestore_split_multiple command does indeed need a restart of the OSD daemon to take effect. Same with the filestore_merge_threshold. These settings also only affect filestore. If you're using bluestore, then they don't mean anything. You can utilize the ceph-objectstore-tool to split

Re: [ceph-users] Ceph cluster network bandwidth?

Another ML thread currently happening is "[ceph-users] Cluster network slower than public network" And It has some good information that might be useful for you. On Thu, Nov 16, 2017 at 10:32 AM David Turner <drakonst...@gmail.com> wrote: > That depends on another questio

Re: [ceph-users] Ceph cluster network bandwidth?

That depends on another question. Does the client write all 3 copies or does the client send the copy to the primary OSD and then the primary OSD sends the write to the secondaries? Someone asked this recently, but I don't recall if an answer was given. I'm not actually certain which is the

Re: [ceph-users] Cluster network slower than public network

There is another thread in the ML right now covering this exact topic. The general consensus is that for most deployments, a separate network for public and cluster is wasted complexity. On Thu, Nov 16, 2017 at 9:59 AM Jake Young wrote: > On Wed, Nov 15, 2017 at 1:07 PM

Re: [ceph-users] Disk Down Emergency

The first step is to make sure that it is out of the cluster. Does `ceph osd stat` show the same number of OSDs as in (it's the same as a line from `ceph status`)? It should show 1 less for up, but if it's still registering the OSD as in then the backfilling won't start. `ceph osd out 0` should

Re: [ceph-users] Reuse pool id

It's probably against the inner workings of Ceph to change the ID of the pool. There are a couple other things in Ceph that keep old data around most likely to prevent potential collisions. One in particular is keeping deleted_snaps in the OSD map indefinitely. One thing I can think of in

Re: [ceph-users] Moving bluestore WAL and DB after bluestore creation

I'm not going to lie. This makes me dislike Bluestore quite a bit. Using multiple OSDs to an SSD journal allowed for you to monitor the write durability of the SSD and replace it without having to out and re-add all of the OSDs on the device. Having to now out and backfill back onto the HDDs is

Re: [ceph-users] Bluestore performance 50% of filestore

me full someday, I’ll try 60GB db > partition – this is the max OSD capacity. > > > > - Rado > > > > *From:* David Turner [mailto:drakonst...@gmail.com] > *Sent:* Tuesday, November 14, 2017 5:38 PM > > > *To:* Milanov, Radoslav Nikiforov <rad...@bu.ed

Re: [ceph-users] Bluestore performance 50% of filestore

203G 0.1731147G 66486 > > kumo-vms3 11 45824M 0.0431147G 11643 > > kumo-volumes3 13 10837M 031147G2724 > > kumo-images3 15 82450M 0.0931147G 10320 > > &g

Re: [ceph-users] S3/Swift :: Pools Ceph

While you can configure 1 pool to be used for RBD and Object storage, I believe that is being deprecated and can cause unforeseen problems in the future. It is definitely not a recommended or common use case. On Tue, Nov 14, 2017 at 4:51 PM Christian Wuerdig < christian.wuer...@gmail.com> wrote:

Re: [ceph-users] Incorrect pool usage statistics

If you know that the pool should be empty, there wouldn't be a problem with piping the ouput of `rados ls` to `rados rm`. By the same notion, if nothing in the pool is needed you can delete the pool and create a new one that will be perfectly empty. On Tue, Nov 14, 2017 at 3:23 PM Karun Josy

Re: [ceph-users] Bluestore performance 50% of filestore

How big was your blocks.db partition for each OSD and what size are your HDDs? Also how full is your cluster? It's possible that your blocks.db partition wasn't large enough to hold the entire db and it had to spill over onto the HDD which would definitely impact performance. On Tue, Nov 14,

Re: [ceph-users] Deleting large pools

ant stuff to go faster. > > Don’t just remove the directory in the filesystem; you need to clean up > the leveldb metadata as well. ;) > Removing the pg via Ceph-objectstore-tool would work fine but I’ve seen > too many people kill the wrong thing to recommend it. > -Greg >

Re: [ceph-users] radosgw multi site different period

I'm assuming you've looked at the period in both places `radosgw-admin period get` and confirmed that the second site is behind the master site (based on epochs). I'm also assuming (since you linked the instructions) that you've done `radosgw-admin period pull` on the second site to get any

Re: [ceph-users] OSD is near full and slow in accessing storage from client

-in reweighting scripts might help your data distribution. reweight-by-utilization On Sun, Nov 12, 2017, 11:41 AM gjprabu <gjpr...@zohocorp.com> wrote: > Hi David, > > Thanks for your valuable reply , once complete the backfilling for new osd > and will consider by increasing

Re: [ceph-users] OSD is near full and slow in accessing storage from client

What's the output of `ceph df` to see if your PG counts are good or not? Like everyone else has said, the space on the original osds can't be expected to free up until the backfill from adding the new osd has finished. You don't have anything in your cluster health to indicate that your cluster

[ceph-users] RGW Multisite replication

Jewel 10.2.7. I have a realm that is not replicating data unless I restart the RGW daemons. It will catch up when I restart the daemon, but then not replicate new information until it's restarted again. This is the only realm with this problem, but all of the realms are configured identically.

Re: [ceph-users] s3 bucket policys

be different. I would name bucket_a and user_a the same name for simplicity so it's obvious which user owns which bucket. On Tue, Nov 7, 2017, 5:25 AM nigel davies <nigdav...@gmail.com> wrote: > Thanks David and All > > I am trying out what you said now. > > When talking to my mana

Re: [ceph-users] s3 bucket policys

advantage of the newer features for rgw. On Mon, Nov 6, 2017 at 11:54 AM nigel davies <nigdav...@gmail.com> wrote: > Thanks all > > David if you can explain how to create subusers with keys i happy to try > and explain to my boss. > > The issue i had with the ACLs, for some rea

Re: [ceph-users] s3 bucket policys

If you don't mind juggling multiple access/secret keys, you can use subusers. Just have 1 user per bucket and create subusers with read, write, etc permissions. The objects are all owned by the 1 user that created the bucket, and then you pass around the subuser keys to the various apps that

[ceph-users] Slack bot

Has anyone developed a bot that can be used in slack to run a few commands against a ceph cluster. I'm thinking about something that could run some read-only commands like `ceph status`. If not, I will be glad to start some work on it. But I figured that I may not be the only person out there

Re: [ceph-users] RAM requirements for OSDs in Luminous

The Ceph docs are versioned. The link you used is for jewel. Change the jewel in the url to luminous to look at the luminous version of the docs. That said, the documentation regarding RAM recommendations has not changed, but this topic was covered fairly recently on the ML. Here is a link to

[ceph-users] Deleting large pools

Jewel 10.2.7; XFS formatted OSDs; no dmcrypt or LVM. I have a pool that I deleted 16 hours ago that accounted for about 70% of the available space on each OSD (averaging 84% full), 370M objects in 8k PGs, ec 4+2 profile. Based on the rate that the OSDs are freeing up space after deleting the

Re: [ceph-users] PGs inconsistent, do I fear data loss?

n the acting group) ... >> backfill would continue to osd 1 of course >> -or data in pg "A" is manually marked as lost, and then continues >> operation from osd 1 's (outdated) copy? >> > > It does deny IO in that case. I think David was pointing out that i

Re: [ceph-users] Slow requests in cache tier with rep_size 2

ere replacing a dozen disks > weekly. > > On the flip side shutting down client access because of a disk failure in > the cluster is *unacceptable* to a product > > On Wed, Nov 1, 2017 at 10:08 AM, David Turner <drakonst...@gmail.com> > wrote: > >> PPS - or min_

Re: [ceph-users] PGs inconsistent, do I fear data loss?

er >> -osd 2 comes back (therefore we have a clean osd in the acting group) ... >> backfill would continue to osd 1 of course >> -or data in pg "A" is manually marked as lost, and then continues >> operation from osd 1 's (outdated) copy? >> > > It

Re: [ceph-users] FAILED assert(p.same_interval_since) and unusable cluster

Jon,     If you are able please test my tentative fix for this issue which is in https://github.com/ceph/ceph/pull/18673 Thanks David On 10/30/17 1:13 AM, Jon Light wrote: Hello, I have three OSDs that are crashing on start with a FAILED assert(p.same_interval_since) error. I ran

Re: [ceph-users] PGs inconsistent, do I fear data loss?

stood better. > The most important thing is that even with min_size=1 writes are > acknowledged after ceph wrote size=2 copies. > In the thread above there is: > > As David already said, when all OSDs are up and in for a PG Ceph will wait > for ALL OSDs to Ack the write. Writes in RADOS ar

[ceph-users] S3 object-size based storage placement policy

ect gateway) Is there some other way to achieve my goal? Thanks in advance, -- David Watzke ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] PGs inconsistent, do I fear data loss?

It looks like you're running with a size = 2 and min_size = 1 (the min_size is a guess, the size is based on how many osds belong to your problem PGs). Here's some good reading for you. https://www.spinics.net/lists/ceph-users/msg32895.html Basically the jist is that when running with size = 2

Re: [ceph-users] Slow requests in cache tier with rep_size 2

PPS - or min_size 1 in production On Wed, Nov 1, 2017 at 10:08 AM David Turner <drakonst...@gmail.com> wrote: > What is your min_size in the cache pool? If your min_size is 2, then the > cluster would block requests to that pool due to it having too few copies > available. > &

Re: [ceph-users] Slow requests in cache tier with rep_size 2

What is your min_size in the cache pool? If your min_size is 2, then the cluster would block requests to that pool due to it having too few copies available. PS - Please don't consider using rep_size 2 in production. On Wed, Nov 1, 2017 at 5:14 AM Eugen Block wrote: > Hi

Re: [ceph-users] Kernel version recommendation

of needing to know all of them or to retrain them. If Ubuntu wasn't stable and secure, it wouldn't be popular. It may not be the most stable or secure, but it sure does get new features faster. On Sat, Oct 28, 2017, 1:01 PM David Turner <drakonst...@gmail.com> wrote: > Saying Ubuntu doe

Re: [ceph-users] Kernel version recommendation

Saying Ubuntu doesn't have a place on servers negates your assertion that the OS is a tool and you should use the right tool for the right job. Sometimes you need an OS that updates its kernel more often than basically never. Back when VMs were gaining traction and CentOS 6 was running the 2.6

Re: [ceph-users] Kernel version recommendation

If you can do an ssh session to the IPMI console and then do that inside of a screen, you can save the output of the screen to a file and look at what was happening on the console when the server locked up. That's how I track kernel panics. On Fri, Oct 27, 2017 at 1:53 PM Bogdan SOLGA

Re: [ceph-users] (no subject)

Your client needs to tell the cluster that the objects have been deleted. '-o discard' is my goto because I'm lazy and it works well enough for me. If you're in need of more performance, then fstrim is the other option. Nothing on the Ceph side can be configured to know when a client no longer

Re: [ceph-users] crush optimize does not work

What does your crush map look like? Also a `ceph df` output. You're optimizing your map for pool #5, if there are other pools with a significant amount of data, then your going to be off on your cluster balance. A big question for balancing a cluster is how big are your PGs? If your primary

Re: [ceph-users] Kernel version recommendation

rbd-nbd is gaining a lot of followers for use as mapping rbds. The kernel driver for RBD's has taken a while to support features of current ceph versions. The nice thing with rbd-nbd is that it has feature parity with the version of ceph you are using and can enable all of the rbd features you

Re: [ceph-users] Speeding up garbage collection in RGW

I had the exact same error when using --bypass-gc. We too decided to destroy this realm and start it fresh. For us, 95% of the data in this realm is backups for other systems and they're find rebuilding it. So our plan is to migrate the 5% of the data to a temporary s3 location and then rebuild

Re: [ceph-users] s3 bucket permishions

Are you talking about RGW buckets with limited permissions for cephx authentication? Or RGW buckets with limited permissions for RGW users? On Wed, Oct 25, 2017 at 12:16 PM nigel davies wrote: > Hay All > > is it possible to set permissions to buckets > > for example if i

Re: [ceph-users] Speeding up garbage collection in RGW

lifecycle work time = 00:01-23:59 > rgw gc max objs = 2647 > rgw lc max objs = 2647 > rgw gc obj min wait = 300 > rgw gc processor period = 600 > rgw gc processor max time = 600 > > > -Ben > > On Tue, Oct 24, 2017 at 9:25 AM, David Turner <drakonst...@gmail.com&g

Re: [ceph-users] Erasure code profile

This can be changed to a failure domain of OSD in which case it could satisfy the criteria. The problem with a failure domain of OSD, is that all of your data could reside on a single host and you could lose access to your data after restarting a single host. On Mon, Oct 23, 2017 at 3:23 PM

Re: [ceph-users] Qs on caches, and cephfs

Multiple cached tiers? 2 tiers to 1 pool or a cache tier to a cache tier? Neither are discussed or mentioned anywhere. At best it might work, but isn't tested for a new release. One cache to multiple pools? Same as above. The luminous docs for cache tiering was updated with "A Word of Caution"

Re: [ceph-users] Looking for help with debugging cephfs snapshots

purged_snaps is persistent indefinitely. If the list gets too large it abbreviates it a bit, but it can cause your osd-map to get a fair bit larger because it keeps track of them. On Sun, Oct 22, 2017 at 10:39 PM Eric Eastman wrote: > On Sun, Oct 22, 2017 at 8:05

Re: [ceph-users] Speeding up garbage collection in RGW

We recently deleted a bucket that was no longer needed that had 400TB of data in it to help as our cluster is getting quite full. That should free up about 30% of our cluster used space, but in the last week we haven't seen nearly a fraction of that free up yet. I left the cluster with this

Re: [ceph-users] Problems with CORS

If you add the external domain to the zonegroup's hostnames and endpoints, then it will be able to respond to that domain. This is assuming that the error message is that the URL is not a valid bucket. We ran into this issue when we upgraded from 10.2.5 to 10.2.9. Any domain used to access RGW

Re: [ceph-users] Check if Snapshots are enabled

Unless you manually issue a snapshot command on the pool, you will never have a snapshot made. But again, I don't think you can disable it. On Fri, Oct 20, 2017, 6:52 AM nigel davies wrote: > ok i have set up an s3 bucket link to my ceph cluster so rgw​,i only > created my

Re: [ceph-users] Check if Snapshots are enabled

I don't know that you can disable snapshots. There isn't an automated method in ceph to run snapshots, but you can easily script it. There are a lot of different types of snapshots in ceph depending if you're using rbd, rgw, or CephFS. There are also caveats and config options you should tweak

<    2   3   4   5   6   7   8   9   10   11   >