Re: [ceph-users] CephFS Quota and ACL support

On Mon, Aug 27, 2018 at 10:53 AM Oliver Freyermuth wrote: > > Thanks for the replies. > > Am 27.08.18 um 19:25 schrieb Patrick Donnelly: > > On Mon, Aug 27, 2018 at 12:51 AM, Oliver Freyermuth > > wrote: > >> These features are critical for us, so right now we use the Fuse client. > >> My hope

Re: [ceph-users] Bluestore crashing constantly with load on newly created cluster/host.

I bumped another post from earlier in the year. I got this reply: Adam Tygart 11:06 PM (8 minutes ago) to me, Kyle, Ceph This issue was related to using Jemalloc. Jemalloc is not as well tested with Bluestore and lead to lots of segfaults. We moved back to the default of tcmalloc with

Re: [ceph-users] Bluestore crashing constantly with load on newly created cluster/host.

Have you created the blockdb partitions or LVM manually ? What size? On 27/08/18 23:48, Tyler Bishop wrote: My host has 256GB of ram.  62GB used under most heavy io workload. _ *Tyler Bishop* EST 2007 O:513-299-7108 x1000 M:513-646-5809

Re: [ceph-users] OSD Segfaults after Bluestore conversion

Okay so far since switching back it looks more stable. I have around 2GB/s and 100k iops flowing with FIO atm to test. _ On Mon, Aug 27, 2018 at 11:06 PM Adam Tygart wrote: > This issue was related to using Jemalloc. Jemalloc is not as well >

Re: [ceph-users] OSD Segfaults after Bluestore conversion

This issue was related to using Jemalloc. Jemalloc is not as well tested with Bluestore and lead to lots of segfaults. We moved back to the default of tcmalloc with Bluestore and these stopped. Check /etc/sysconfig/ceph under RHEL based distros. -- Adam On Mon, Aug 27, 2018 at 9:51 PM Tyler

Re: [ceph-users] OSD Segfaults after Bluestore conversion

Did you solve this? Similar issue. _ On Wed, Feb 28, 2018 at 3:46 PM Kyle Hutson wrote: > I'm following up from awhile ago. I don't think this is the same bug. The > bug referenced shows "abort: Corruption: block checksum mismatch", and I'm > not

Re: [ceph-users] Bluestore crashing constantly with load on newly created cluster/host.

My host has 256GB of ram. 62GB used under most heavy io workload. _ *Tyler Bishop* EST 2007 O: 513-299-7108 x1000 M: 513-646-5809 http://BeyondHosting.net This email is intended only for the recipient(s) above and/or

Re: [ceph-users] Bluestore crashing constantly with load on newly created cluster/host.

I had blockdb in ssd, with 3 OSDs per host (8G ram) and the default 3G bluestore_cache_size_ssd I stopped having inconsistencies dropping the cache to 1G. On 27/08/18 23:32, Tyler Bishop wrote: Having a constant segfault issue under io load with my newly created bluestore deployment.

[ceph-users] Bluestore crashing constantly with load on newly created cluster/host.

Having a constant segfault issue under io load with my newly created bluestore deployment. https://pastebin.com/82YjXRm7 Setup is 28GB SSD LVM for block.db and 6T spinner for data. Config: [global] fsid = REDACTED mon_initial_members = cephmon-1001, cephmon-1002, cephmon-1003 mon_host =

Re: [ceph-users] fixable inconsistencies but more appears

Well. it seems memory I have 3 ODS per host with 8G RAM and block.db in SSD Setting bluestore_cache_size_ssd=1G seems to have fixed the problem. No new inconsistencies. On 21/08/18 16:09, Paul Emmerich wrote: Are you running tight on memory? Paul 2018-08-21 20:37 GMT+02:00 Alfredo

Re: [ceph-users] CephFS Quota and ACL support

+Ilya On Mon, Aug 27, 2018 at 10:53 AM Oliver Freyermuth < freyerm...@physik.uni-bonn.de> wrote: > Thanks for the replies. > > Am 27.08.18 um 19:25 schrieb Patrick Donnelly: > > On Mon, Aug 27, 2018 at 12:51 AM, Oliver Freyermuth > > wrote: > >> These features are critical for us, so right now

Re: [ceph-users] CephFS Quota and ACL support

Thanks for the replies. Am 27.08.18 um 19:25 schrieb Patrick Donnelly: > On Mon, Aug 27, 2018 at 12:51 AM, Oliver Freyermuth > wrote: >> These features are critical for us, so right now we use the Fuse client. My >> hope is CentOS 8 will use a recent enough kernel >> to get those features

Re: [ceph-users] CephFS Quota and ACL support

On Mon, Aug 27, 2018 at 12:51 AM, Oliver Freyermuth wrote: > These features are critical for us, so right now we use the Fuse client. My > hope is CentOS 8 will use a recent enough kernel > to get those features automatically, though. Your cluster needs to be running Mimic and Linux v4.17+.

Re: [ceph-users] Odp.: pgs incomplete and inactive

I came across a problem like this before with small flash OSDs for metadata. There is an open tracker about why it was able to fill 100% of the way up, but no work done on it in 6 months after I got back to healthy. The way I did that was deleting one copy of a PG from each OSD (different PGs on

Re: [ceph-users] ceph-fuse slow cache?

On Mon, Aug 27, 2018 at 4:47 AM Stefan Kooman wrote: > > Hi, > > Quoting Yan, Zheng (uker...@gmail.com): > > Could you strace apacha process, check which syscall waits for a long time. > > Yes, that's how I did all the tests (strace -t -T apache2 -X). With > debug=20 (ceph-fuse) you see apache

Re: [ceph-users] cephfs kernel client hangs

On Mon, Aug 27, 2018 at 6:10 AM Zhenshi Zhou wrote: > > Hi, > The kernel version is 4.12.8-1.el7.elrepo.x86_64. > Client.267792 has gone as I restart the server at weekend. > Does ceph-fuse more stable than kernel client? > For old kernels such as 4.12. ceph-fuse is more stable. If you use

Re: [ceph-users] limited disk slots - should I ran OS on SD card ?

This exact problem with the OS disk and problems deploying lots of servers in an efficient way was the main motivator for developing our croit orchestration product: https://croit.io I've talked about this on a few Ceph days, but the short summary is: We started with Ceph in 2013 and decided to

Re: [ceph-users] RBD image "lightweight snapshots"

On Mon, Aug 27, 2018 at 3:29 AM Bartosz Rabiega wrote: > > Bumping the topic. > > > So, what do you think guys? Not sure if you saw my response from August 13th, but I stated that this is something that you should be able to build right now using the RADOS Python bindings and the rbd CLI. It

Re: [ceph-users] cephfs kernel client hangs

Hi, The kernel version is 4.12.8-1.el7.elrepo.x86_64. Client.267792 has gone as I restart the server at weekend. Does ceph-fuse more stable than kernel client? Yan, Zheng 于2018年8月27日周一 上午11:41写道： > please check client.213528, instead of client.267792. which version of > kernel client.213528

Re: [ceph-users] Why rbd rn did not clean used pool?

On Sat, Aug 25, 2018 at 10:29 AM Fyodor Ustinov wrote: > > Hi! > > Configuration: > rbd - erasure pool > rbdtier - tier pool for rbd > > ceph osd tier add-cache rbd rbdtier 549755813888 > ceph osd tier cache-mode rbdtier writeback > > Create new rbd block device: > rbd create --size 16G rbdtest

Re: [ceph-users] ceph-fuse slow cache?

Hi, Quoting Yan, Zheng (uker...@gmail.com): > Could you strace apacha process, check which syscall waits for a long time. Yes, that's how I did all the tests (strace -t -T apache2 -X). With debug=20 (ceph-fuse) you see apache waiting for almost 20 seconds before it starts serving data:

[ceph-users] Odp.: pgs incomplete and inactive

Hello Josef, I would suggest setting up a bigger disk (if not physical then maybe a LVM volume from 2 smaller disks) and cloning (remember about extended attributes!) the OSD data dir to the new disk, then try to bring the OSD back into cluster. -- Tomasz Kuzemko tomasz.kuze...@corp.ovh.com

Re: [ceph-users] pgs incomplete and inactive

The fullratio was ignored, that's why that happenned most likely. I can't delete pgs, because it's only kb's worth of space - the osd is 40gb, 39.8 gb is taken up by omap - that's why i can't move/extract. Any clue on how to compress/move away the omap dir? On 27/08/18 12:34, Paul Emmerich

Re: [ceph-users] pgs incomplete and inactive

Don't ever let an OSD run 100% full, that's usually bad news. Two ways to salvage this: 1. You can try to extract the PGs with ceph-objectstore-tool and inject them into another OSD; Ceph will find them and recover 2. You seem to be using Filestore, so you should easily be able to just delete a

Re: [ceph-users] CephFS Quota and ACL support

It is supported in mainline kernel from elrepo. http://elrepo.org/tiki/tiki-index.php > On 27.08.2018, at 10:51, Oliver Freyermuth > wrote: > > Dear Cephalopodians, > > sorry if this is the wrong place to ask - but does somebody know if the > recently

Re: [ceph-users] Error EINVAL: (22) Invalid argument While using ceph osd safe-to-destroy

Hi, could you please paste your osd tree and the exact command you try to execute? Extra note, the while loop in the instructions look like it's bad. I had to change it to make it work in bash. The documented command didn't work for me either. Regards, Eugen Zitat von Robert Stanford :

[ceph-users] pgs incomplete and inactive

Hi, i've had a very ugly thing happen to me over the weekend. Some of my  OSDs in a root that handles metadata pools overflowed to 100% disk usage due to omap size(even though i had 97% full ratio, which is odd) and refused to start. There were some pgs on those OSDs that went away with them.

Re: [ceph-users] Design a PetaByte scale CEPH object storage

James, I would recommend that you do the following a) write out a clear set of requirements and use cases for this system. Do not mention any specific technology b) plan to install and test a small ProofOfConcept system. You can then assess if it meets the requirement in (a) On Mon, 27 Aug 2018

Re: [ceph-users] Ceph-Deploy error on 15/71 stage

Hi Jones, all ceph logs are in the directory /var/log/ceph/, each daemon has its own log file, e.g. OSD logs are named ceph-osd.*. I haven't tried it but I don't think SUSE Enterprise Storage deploys OSDs on partitioned disks. Is there a way to attach a second disk to the OSD nodes,

[ceph-users] CephFS Quota and ACL support

Dear Cephalopodians, sorry if this is the wrong place to ask - but does somebody know if the recently added quota support in the kernel client, and the ACL support, are going to be backported to RHEL 7 / CentOS 7 kernels? Or can someone redirect me to the correct place to ask? We don't have a

Re: [ceph-users] Design a PetaByte scale CEPH object storage

> I am a software developer and am new to this domain. So maybe first get some senior system admin or so? You also do not want me to start doing some amateur brain surgery, do you? > each file has approx 15 TB Pfff, maybe rethink/work this to -Original Message- From:

Re: [ceph-users] RBD image "lightweight snapshots"

Bumping the topic. So, what do you think guys? On 08/13/2018 12:22 PM, Bartosz Rabiega wrote: On 08/11/2018 07:56 AM, Paweł Sadowski wrote: On 08/10/2018 06:24 PM, Gregory Farnum wrote: On Fri, Aug 10, 2018 at 4:53 AM, Paweł Sadowsk wrote: On 08/09/2018 04:39 PM, Alex Elder wrote: On