Re: [ceph-users] pgs stuck inactive

[root@storage2 ~]# gdb -ex 'r' -ex 't a a bt full' -ex 'q' --args ceph-objectstore-tool import-rados volumes pg.3.367.export.OSD.35 GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-94.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later

Re: [ceph-users] mkjournal error creating journal ... : (13) Permission denied

Thank you so much Peter. The 'udevadm trigger' after 'partprobe' triggered the udev rules and I've found out that even before the udev ruleset triggers the owner is already ceph:ceph. I've dug into ceph-disk a little more and found out that there is a symbolic link of

Re: [ceph-users] pgs stuck inactive

Can you install the debuginfo for ceph (how this works depends on your distro) and run the following? # gdb -ex 'r' -ex 't a a bt full' -ex 'q' --args ceph-objectstore-tool import-rados volumes pg.3.367.export.OSD.35 On Thu, Mar 16, 2017 at 12:02 AM, Laszlo Budai wrote:

Re: [ceph-users] Ceph Cluster Failures

On Thu, Mar 16, 2017 at 02:22:08AM +, Rich Rocque wrote: > Has anyone else run into this or have any suggestions on how to remedy it? We need a LOT more info. > After a couple months of almost no issues, our Ceph cluster has > started to have frequent failures. Just this week it's failed

Re: [ceph-users] Moving data from EC pool with replicated cache tier to replicated pool

On Tue, Mar 14, 2017 at 6:20 AM pwoszuk wrote: > Hi all > > I need a help with operation of moving all data from one pool to another. > > pool1: ECpool with replicated cache tier pool (name it: pool1a) > > pool2: replicated pool > > need to move data from pool1 -> pool2 >

[ceph-users] Ceph Cluster Failures

Hi, After a couple months of almost no issues, our Ceph cluster has started to have frequent failures. Just this week it's failed about three times. The issue appears to be than an MDS or Monitor will fail and then all clients hang. After that, all clients need to be forcibly restarted.

Re: [ceph-users] total storage size available in my CEPH setup?

Hello, On Wed, 15 Mar 2017 21:36:00 + James Okken wrote: > Thanks gentlemen, > > I hope to add more OSD since we will need a good deal more than 2.3TB and I > fo want to leave free space / margins. > > I am also thinking of reducing the replication to2 . > I am sure I can google how to

Re: [ceph-users] Odd latency numbers

Hello, On Wed, 15 Mar 2017 16:49:00 + Rhian Resnick wrote: > Morning all, > > > We starting to apply load to our test cephfs system and are noticing some odd > latency numbers. We are using erasure coding for the cold data pools and > replication for our our cache tiers (not on ssd yet)

Re: [ceph-users] Directly addressing files on individual OSD

As I parse Youssef’s message, I believe there are some misconceptions. It might help if you could give a bit more info on what your existing ‘cluster’ is running. NFS? CIFS/SMB? Something else? 1) Ceph regularly runs scrubs to ensure that all copies of data are consistent. The checksumming

Re: [ceph-users] total storage size available in my CEPH setup?

Thanks gentlemen, I hope to add more OSD since we will need a good deal more than 2.3TB and I fo want to leave free space / margins. I am also thinking of reducing the replication to2 . I am sure I can google how to do that. But I am sure most of my results are going to be people telling me

[ceph-users] Ceph Tech Talk next Thurs

Hey cephers, Just a reminder that we'll be having our monthly Ceph Tech Talk next Thursday at 1p EST. http://ceph.com/ceph-tech-talks/ Chris Holcombe from Canonical will be talking about some of the work they have been doing to streamline Ceph Deployment as well as a walkthrough on writing

Re: [ceph-users] Creating Ceph Pools on different OSD's -- crushmap ?

Ok, I found this tutorial on crushmap from han. Hopefully I should get my structure accomplished using crushmap. https://www.sebastien-han.fr/blog/2012/12/07/ceph-2-speed-storage-with-crush/ -- Deepak From: Deepak Naidu Sent: Wednesday, March 15, 2017 12:45 PM To: ceph-users Subject: Creating

Re: [ceph-users] Creating new Pools - PG's

Hi David, Thank you for your response! I was thinking that I may use Ceph to back other projects outside of our infrastructure, so I calculated 75% VM and 25% other usage when I created the pool. Cheers, Mike On Wed, Mar 15, 2017 at 12:57 PM, David Turner wrote: >

Re: [ceph-users] Creating new Pools - PG's

Especially if you are planning to remove the old pool after you migrate, you shouldn't have any problems with this plan. If you were going to leave both running indefinitely, then I'd recommend calculating out how many PGs you should add based on how many OSDs you have. Based on your numbers,

[ceph-users] Creating Ceph Pools on different OSD's -- crushmap ?

Hello, I am trying to address the failure domain & performance/isolation of pools based on what OSD they can belong to. Let me give example. Can I achieve this with crurshmap ruleset or any other method, if so how? Example: 10x storage servers each have 3x OSD ie OSD.0 through OSD29 --

[ceph-users] ceph 0.94.10 ceph-objectstore-tool segfault

Hello, I'm trying to do an import-rados operation, but the ceph-objectstore-tool crashes with segfault: [root@storage1 ~]# ceph-objectstore-tool import-rados images pg6.6exp-osd1 *** Caught signal (Segmentation fault) ** in thread 7f84e0b24880 ceph version 0.94.10

Re: [ceph-users] Ceph-deploy and git.ceph.com

Thanks for all the help so far. Just to be clear…if I am planning on upgrading the cluster from Hammer in say the next 3 months…what is the suggested upgrade path? Thanks again, Shain > On Mar 15, 2017, at 2:05 PM, Abhishek Lekshmanan wrote: > > > > On 15/03/17 18:32,

Re: [ceph-users] Ceph-deploy and git.ceph.com

On 15/03/17 18:32, Shinobu Kinjo wrote: So description of Jewel is wrong? http://docs.ceph.com/docs/master/releases/ Yeah we missed updating jewel dates as well when updating about hammer, Jewel is an LTS and would get more upgrades. Once Luminous is released, however, we'll eventually

Re: [ceph-users] Ceph-deploy and git.ceph.com

So description of Jewel is wrong? http://docs.ceph.com/docs/master/releases/ On Thu, Mar 16, 2017 at 2:27 AM, John Spray wrote: > On Wed, Mar 15, 2017 at 5:04 PM, Shinobu Kinjo wrote: >> It may be probably kind of challenge but please consider Kraken (or

Re: [ceph-users] Ceph-deploy and git.ceph.com

On Wed, Mar 15, 2017 at 5:04 PM, Shinobu Kinjo wrote: > It may be probably kind of challenge but please consider Kraken (or > later) because Jewel will be retired: > > http://docs.ceph.com/docs/master/releases/ Nope, Jewel is LTS, Kraken is not. Kraken will only receive

Re: [ceph-users] Ceph-deploy and git.ceph.com

Would you file this as a doc bug? So we discuss properly with tracking. http://tracker.ceph.com On Thu, Mar 16, 2017 at 2:17 AM, Deepak Naidu wrote: >>> because Jewel will be retired: > Hmm. Isn't Jewel LTS ? > > Every other stable releases is a LTS (Long Term Stable) and

Re: [ceph-users] Ceph-deploy and git.ceph.com

>> because Jewel will be retired: Hmm. Isn't Jewel LTS ? Every other stable releases is a LTS (Long Term Stable) and will receive updates until two LTS are published. -- Deepak > On Mar 15, 2017, at 10:09 AM, Shinobu Kinjo wrote: > > It may be probably kind of challenge

Re: [ceph-users] Ceph-deploy and git.ceph.com

It may be probably kind of challenge but please consider Kraken (or later) because Jewel will be retired: http://docs.ceph.com/docs/master/releases/ On Thu, Mar 16, 2017 at 1:48 AM, Shain Miley wrote: > No this is a production cluster that I have not had a chance to upgrade yet.

[ceph-users] Odd latency numbers

Morning all, We starting to apply load to our test cephfs system and are noticing some odd latency numbers. We are using erasure coding for the cold data pools and replication for our our cache tiers (not on ssd yet) . We noticed the following high latency on one node and it seams to be

Re: [ceph-users] Ceph-deploy and git.ceph.com

No this is a production cluster that I have not had a chance to upgrade yet. We had an is with the OS on a node so I am just trying to reinstall ceph and hope that the osd data is still in tact. Once I get things stable again I was planning on upgrading…but the upgrade is a bit intensive by

Re: [ceph-users] Ceph-deploy and git.ceph.com

Just curious, why you still want to deploy new hammer instead of stable jewel? Is this a test environment? the last .10 release was basically for bug fixes for 0.94.9. On Wed, Mar 15, 2017 at 9:16 AM, Shinobu Kinjo wrote: > FYI: >

Re: [ceph-users] Ceph-deploy and git.ceph.com

I had similar issue when using older version of ceph-deploy. I see the URL got.ceph.com doesn't work on browser as well. To resolve this, I installed the latest version of ceph-deploy and it worked fine. New version wasn't using git.ceph.com. During ceph-deploy you can mention what version of

Re: [ceph-users] Ceph-deploy and git.ceph.com

FYI: https://plus.google.com/+Cephstorage/posts/HuCaTi7Egg3 On Thu, Mar 16, 2017 at 1:05 AM, Shain Miley wrote: > Hello, > I am trying to deploy ceph to a new server using ceph-deply which I have > done in the past many times without issue. > > Right now I am seeing a timeout

[ceph-users] Ceph-deploy and git.ceph.com

Hello, I am trying to deploy ceph to a new server using ceph-deply which I have done in the past many times without issue. Right now I am seeing a timeout trying to connect to git.ceph.com : [hqosd6][INFO ] Running command: env DEBIAN_FRONTEND=noninteractive apt-get -q

Re: [ceph-users] Log message --> "bdev(/var/lib/ceph/osd/ceph-x/block) aio_submit retries"

On Wed, 15 Mar 2017, Brad Hubbard wrote: > +ceph-devel > > On Wed, Mar 15, 2017 at 5:25 PM, nokia ceph wrote: > > Hello, > > > > We suspect these messages not only at the time of OSD creation. But in idle > > conditions also. May I know what is the impact of these

Re: [ceph-users] pgs stuck inactive

Hello, the ceph-objectstore-tool import-rados volumes pg.3.367.export.OSD.35 command crashes. ~# ceph-objectstore-tool import-rados volumes pg.3.367.export.OSD.35 *** Caught signal (Segmentation fault) ** in thread 7f85b60e28c0 ceph version 0.94.10 (b1e0532418e4631af01acbc0cedd426f1905f4af)

Re: [ceph-users] [ceph-fuse] Quota size change does not notify another ceph-fuse client.

On Wed, Mar 15, 2017 at 1:53 AM, yu2xiangyang wrote: > Dear cephers, > > I met a problem when using ceph-fuse with quota enabled. > > My ceph version is : > > ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367) . > > I have two ceph-fuse process in two different

Re: [ceph-users] pgs stuck inactive

Ok. Delete the dirs using the ceph-objectstore-tool. DONE ceph pg force_create_pg 3.367 lead me to this state: HEALTH_WARN 1 pgs stuck inactive; 1 pgs stuck unclean; 16 requests are blocked > 32 sec; 2 osds have slow requests; noout flag(s) set pg 3.367 is stuck inactive since forever, current

Re: [ceph-users] pgs stuck inactive

Hello, So, I've done the following seps: 1. set noout 2. stop osd2 3. ceph-objectstore-tool remove 4. start osd2 5. repeat step 2-4 on osd 28 and 35 then I've run the ceph pg force_create_pg 3.367. This has left the PG in creating state: # ceph -s cluster

[ceph-users] 答复: 答复: Pipe "deadlock" in Hammer, 0.94.5

Hi, sir. I'm sorry I made a mistake, the fix that you provided should be the one we need, is it safe for us to simply "git cherry-pick" that commit into our 0.94.5 version? So sorry for my mistake. Thank you. On Wed, Jan 11, 2017 at 3:59 PM, 许雪寒 wrote: > In our test, when

Re: [ceph-users] mkjournal error creating journal ... : (13) Permission denied

On 03/15/17 08:43, Gunwoo Gim wrote: > After a reboot, all the partitions of LVM don't show up in > /dev/mapper -nor in the /dev/dm- or /proc/partitions- though > the whole disks show up; I have to make the hosts run one 'partprobe' > every time they boot so as to have the partitions all show up.

Re: [ceph-users] Ceph Bluestore

Hello, On Wed, 15 Mar 2017 09:07:10 +0100 Michał Chybowski wrote: > > Hello, > > > > your subject line has little relevance to your rather broad questions. > > > > On Tue, 14 Mar 2017 23:45:26 +0100 Michał Chybowski wrote: > > > >> Hi, > >> > >> I'm going to set up a small cluster (5 nodes

Re: [ceph-users] Log message --> "bdev(/var/lib/ceph/osd/ceph-x/block) aio_submit retries"

+ceph-devel On Wed, Mar 15, 2017 at 5:25 PM, nokia ceph wrote: > Hello, > > We suspect these messages not only at the time of OSD creation. But in idle > conditions also. May I know what is the impact of these error? Can we safely > ignore this? Or is there any

Re: [ceph-users] Ceph Bluestore

W dniu 15.03.2017 o 09:05, Eneko Lacunza pisze: Hi Michal, El 14/03/17 a las 23:45, Michał Chybowski escribió: I'm going to set up a small cluster (5 nodes with 3 MONs, 2 - 4 HDDs per node) to test if ceph in such small scale is going to perform good enough to put it into production

Re: [ceph-users] Ceph Bluestore

Hello, your subject line has little relevance to your rather broad questions. On Tue, 14 Mar 2017 23:45:26 +0100 Michał Chybowski wrote: Hi, I'm going to set up a small cluster (5 nodes with 3 MONs, 2 - 4 HDDs per node) to test if ceph in such small scale is going to perform good enough to

Re: [ceph-users] Ceph Bluestore

Hi Michal, El 14/03/17 a las 23:45, Michał Chybowski escribió: I'm going to set up a small cluster (5 nodes with 3 MONs, 2 - 4 HDDs per node) to test if ceph in such small scale is going to perform good enough to put it into production environment (or does it perform well only if there are

Re: [ceph-users] mkjournal error creating journal ... : (13) Permission denied

After a reboot, all the partitions of LVM don't show up in /dev/mapper -nor in the /dev/dm- or /proc/partitions- though the whole disks show up; I have to make the hosts run one 'partprobe' every time they boot so as to have the partitions all show up. I've found out that the udev rules have

Re: [ceph-users] osd_disk_thread_ioprio_priority help

On Wed, Mar 15, 2017 at 2:41 AM, Alex Gorbachev wrote: > On Mon, Mar 13, 2017 at 6:09 AM, Florian Haas wrote: >> On Mon, Mar 13, 2017 at 11:00 AM, Dan van der Ster >> wrote: I'm sorry, I may have worded that in a manner

Re: [ceph-users] Log message --> "bdev(/var/lib/ceph/osd/ceph-x/block) aio_submit retries"

Hello, We suspect these messages not only at the time of OSD creation. But in idle conditions also. May I know what is the impact of these error? Can we safely ignore this? Or is there any way/config to fix this problem Few occurrence for these events as follows:--- 2017-03-14

[ceph-users] Directly addressing files on individual OSD

We currently run a commodity cluster that supports a few petabytes of data. Each node in the cluster has 4 drives, currently mounted as /0 through /3. We have been researching alternatives for managing the storage, Ceph being one possibility, iRODS being another. For preservation purposes, we