On Fri, 5 Sep 2014 13:46:17 +0800 Ding Dinghua wrote:
> 2014-09-05 13:19 GMT+08:00 Christian Balzer :
>
> >
> > Hello,
> >
> > On Fri, 5 Sep 2014 12:09:11 +0800 Ding Dinghua wrote:
> >
> > > Please see my comment below:
> > >
> > >
> > > 2014-09-04 21:33 GMT+08:00 Christian Balzer :
> > >
> > > >
Hi,
Sorry for the lack of information yesterday, this was "solved" after some 30
minutes, after having reloaded/restarted all osd daemons.
Unfortunately we couldn’t pin point it to a single OSD or drive, all drives
seemed ok, some had a bit higher latency and we tried to out / in them to see
if
I was waiting for the schedule, topics seem to be interesting.
I'm going to register now :)
BTW, are the speeches in french or english? (As I see loic,sebastian and yann
as speakers)
- Mail original -
De: "Patrick McGarry"
À: "Ceph Devel" , "Ceph-User"
, "Ceph User Committee"
Envoyé
Solved.
ceph-authtool /tmp/ceph.mon.keyring --import-keyring
/etc/ceph/ceph.client.admin.keyring
ceph-authtool /tmp/ceph.mon.keyring --import-keyring
/etc/ceph/ceph.client.admin.keyring
The above two steps on mon node solved the issue.
Thanks!
Regards,
santhosh
On Fri, Sep 5, 2014 at 11:05
2014-09-05 13:19 GMT+08:00 Christian Balzer :
>
> Hello,
>
> On Fri, 5 Sep 2014 12:09:11 +0800 Ding Dinghua wrote:
>
> > Please see my comment below:
> >
> >
> > 2014-09-04 21:33 GMT+08:00 Christian Balzer :
> >
> > >
> > > Hello,
> > >
> > > On Thu, 4 Sep 2014 20:56:31 +0800 Ding Dinghua wrote:
>
Thanks Sahana!
That helped and now it works on OSD's and objectgw nodes and not working
the same trick on mon node.
Regards,
Santhosh
On Fri, Sep 5, 2014 at 11:01 AM, Sahana Lokeshappa <
sahana.lokesha...@sandisk.com> wrote:
> HI Santhosh,
>
>
>
> Copy updated ceph.conf and keyrings from admi
HI Santhosh,
Copy updated ceph.conf and keyrings from admin node to all cluster nodes
(present in /etc/ceph/) . If you are using ceph-deploy , use this command from
admin node.
ceph-deploy –overwrite-conf admin cluster-node1 cluster-node2
Sahana Lokeshappa
Test Development Engineer I
SanDisk C
Just echoing what Christian said.
Also, iirc the "currently waiting for subobs on [" could also mean a
problem on those as it waits for ack from them (I might remember wrong).
If that is the case you might want to check in on osd 13 & 37 as well.
With the cluster load and size you should not hav
On Fri, Sep 5, 2014 at 8:42 AM, James Devine wrote:
> I'm using 3.13.0-35-generic on Ubuntu 14.04.1
>
Was there any kernel message when the hang happened? We have fixed a
few bugs since 3.13 kernel, please use 3.16 kernel if possible.
Yan, Zheng
>
> On Thu, Sep 4, 2014 at 6:08 PM, Yan, Zheng
Hi All,
I am trying to configure Ceph with 2 OSD, one MON, One ADMIN, and One
ObjectGW nodes.
My admin node gives proper output for command ceph -s and on other ceph
nodes gives me similar output below.
2014-09-05 10:45:01.946215 7f45d8852700 -1 monclient(hunting): ERROR:
missing keyring, canno
On Thu, Sep 4, 2014 at 10:23 PM, Dan van der Ster wrote:
> Hi Martin,
>
> September 4 2014 10:07 PM, "Martin B Nielsen" wrote:
> > Hi Dan,
> >
> > We took a different approach (and our cluster is tiny compared to many
> others) - we have two pools;
> > normal and ssd.
> >
> > We use 14 disks in
Hello,
On Fri, 5 Sep 2014 12:09:11 +0800 Ding Dinghua wrote:
> Please see my comment below:
>
>
> 2014-09-04 21:33 GMT+08:00 Christian Balzer :
>
> >
> > Hello,
> >
> > On Thu, 4 Sep 2014 20:56:31 +0800 Ding Dinghua wrote:
> >
> > Aside from what Loic wrote, why not replace the network contro
ceph -w output:
health HEALTH_WARN 441 pgs degraded; 441 pgs stuck unclean; recovery
131518/1036770 objects degraded (12.685%); 4/31 in osds are down; noout
flag(s) set
2014-09-05 11:36 GMT+08:00 Jason King :
> Hi,
>
> What's the status of your cluster after the node failure?
>
> Jason
>
>
> 20
Thanks for reply.
The new node is more powerful than the broken one, so this is a
hardware-upgrade process too, and I think replace an node may be a common
operation, so I want to explore a general-purpose method.
2014-09-04 21:15 GMT+08:00 Loic Dachary :
> Hi,
>
> If the new machine can host th
Please see my comment below:
2014-09-04 21:33 GMT+08:00 Christian Balzer :
>
> Hello,
>
> On Thu, 4 Sep 2014 20:56:31 +0800 Ding Dinghua wrote:
>
> Aside from what Loic wrote, why not replace the network controller or if
> it is onboard, add a card?
>
> > Hi all,
> > I'm new to ceph, and
getting below error:
[nk21l01si-d01-ceph001][INFO ] Running command: sudo ceph-disk -v activate
--mark-init sysvinit --mount /var/local/osd0
[nk21l01si-d01-ceph001][WARNIN] DEBUG:ceph-disk:Cluster uuid is
08985bbc-5a98-4614-9267-3e0a91e7358b
[nk21l01si-d01-ceph001][WARNIN] INFO:ceph-disk:Runnin
Hi,
What's the status of your cluster after the node failure?
Jason
2014-09-04 21:33 GMT+08:00 Christian Balzer :
>
> Hello,
>
> On Thu, 4 Sep 2014 20:56:31 +0800 Ding Dinghua wrote:
>
> Aside from what Loic wrote, why not replace the network controller or if
> it is onboard, add a card?
>
> >
I??tested osd with btrfs, looks like the write performance is a little bit
faste than xfs, but there are a lot of painful problem like:
1. restart osd take about 1 - 2 hours
2. osd will down everday
3. etc.
-- --
??: "Andrew Thrift";;
: 20
I have recently been wondering the same thing.
Does anyone have any experience with this ?
On Fri, Sep 5, 2014 at 12:18 AM, Andrei Mikhailovsky
wrote:
> Hello guys,
>
> I was wondering if there is a benefit of using journal-less btrfs file
> system on the cache pool osds? Would it speed up the
Hello,
On Thu, 4 Sep 2014 14:49:39 -0700 Craig Lewis wrote:
> On Thu, Sep 4, 2014 at 9:21 AM, Dan Van Der Ster
> wrote:
>
> >
> >
> > 1) How often are DC S3700's failing in your deployments?
> >
>
> None of mine have failed yet. I am planning to monitor the wear level
> indicator, and preemp
I'm using 3.13.0-35-generic on Ubuntu 14.04.1
On Thu, Sep 4, 2014 at 6:08 PM, Yan, Zheng wrote:
> On Fri, Sep 5, 2014 at 3:24 AM, James Devine wrote:
> > It took a week to happen again, I had hopes that it was fixed but alas
> it is
> > not. Looking at top logs on the active mds server, the l
On Thu, 4 Sep 2014 12:02:13 +0200 David wrote:
> Hi,
>
> We’re running a ceph cluster with version:
>
> 0.67.7-1~bpo70+1
>
> All of a sudden we’re having issues with the cluster (running RBD images
> for kvm) with slow requests on all of the OSD servers. Any idea why and
> how to fix it?
>
You
Hello,
On Thu, 4 Sep 2014 22:51:35 + Zojonc, Josh wrote:
> I’m trying to figure if its possible to compute how many nodes I can
> lose. I’ve got a cluster of 6 boxes with 9 disks each for a total of 54
> OSD’s. There are a total of 1800 placement groups on those OSD’s. The
> replica size
On Fri, Sep 5, 2014 at 3:24 AM, James Devine wrote:
> It took a week to happen again, I had hopes that it was fixed but alas it is
> not. Looking at top logs on the active mds server, the load average was
> 0.00 the whole time and memory usage never changed much, it is using close
> to 100% and s
I’m trying to figure if its possible to compute how many nodes I can lose.
I’ve got a cluster of 6 boxes with 9 disks each for a total of 54 OSD’s.
There are a total of 1800 placement groups on those OSD’s. The replica size is
3.
Is there a way to figure it out or is more information needed
On 05/09/14 10:05, Dan van der Ster wrote:
That's good to know. I would plan similarly for the wear out. But I want to
also prepare for catastrophic failures -- in the past we've had SSDs just
disappear like a device unplug. Those were older OCZ's though...
Yes - the Intel dc style drives s
Hi,
We’re running a ceph cluster with version:
0.67.7-1~bpo70+1
All of a sudden we’re having issues with the cluster (running RBD images for
kvm) with slow requests on all of the OSD servers.
Any idea why and how to fix it?
2014-09-04 11:56:35.868521 mon.0 [INF] pgmap v12504451: 6860 pgs: 686
Hi Craig,
September 4 2014 11:50 PM, "Craig Lewis" wrote:
> On Thu, Sep 4, 2014 at 9:21 AM, Dan Van Der Ster
> wrote:
>
>> 1) How often are DC S3700's failing in your deployments?
>
> None of mine have failed yet. I am planning to monitor the wear level
> indicator, and preemptively
> repl
On Thu, Sep 4, 2014 at 9:21 AM, Dan Van Der Ster
wrote:
>
>
> 1) How often are DC S3700's failing in your deployments?
>
None of mine have failed yet. I am planning to monitor the wear level
indicator, and preemptively replace any SSDs that go below 10%. Manually
flushing the journal, replacin
Hi Martin,
September 4 2014 10:07 PM, "Martin B Nielsen" wrote:
> Hi Dan,
>
> We took a different approach (and our cluster is tiny compared to many
> others) - we have two pools;
> normal and ssd.
>
> We use 14 disks in each osd-server; 8 platter and 4 ssd for ceph, and 2 ssd
> for OS/journ
Hi Dan,
We took a different approach (and our cluster is tiny compared to many
others) - we have two pools; normal and ssd.
We use 14 disks in each osd-server; 8 platter and 4 ssd for ceph, and 2 ssd
for OS/journals. We partitioned the two OS ssd as raid1 using about half
the space for the OS and
Is there a tool to show a the layout of PGs in a cluster (which OSD/node that
duplicate PGs are placed in? Something like a table with the PG number on one
side, and columns representing nodes/OSDs, with the OSD containing a PG filled
in/marked?
--
Hi Stefan,
September 4 2014 9:13 PM, "Stefan Priebe" wrote:
> Hi Dan, hi Robert,
>
> Am 04.09.2014 21:09, schrieb Dan van der Ster:
>
>> Thanks again for all of your input. I agree with your assessment -- in
>> our cluster we avg <3ms for a random (hot) 4k read already, but > 40ms
>> for a 4k
It took a week to happen again, I had hopes that it was fixed but alas it
is not. Looking at top logs on the active mds server, the load average was
0.00 the whole time and memory usage never changed much, it is using close
to 100% and some swap but since I changed memory.swappiness swap usage
has
This is good to know. I just recompiled the CentOS7 3.10 kernel to enable
bcache (I doubt they patched bcache since they don't compile/enable it).
I've seen when I ran Ceph in VMs on my workstation that there were oops
with bcache, but doing the bcache device and the backend device even with
two co
On Thu, Sep 4, 2014 at 12:45 AM, Maros Vegh wrote:
> The ceph fs is mounted via the kernel client.
> The clients are running on this kernel:
> 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u1 x86_64 GNU/Linux
3.2 is a pretty old kernel. Can you try it on a newer kernel, say
3.13 or 3.14? Is there an
Hi Dan, hi Robert,
Am 04.09.2014 21:09, schrieb Dan van der Ster:
Thanks again for all of your input. I agree with your assessment -- in
our cluster we avg <3ms for a random (hot) 4k read already, but > 40ms
for a 4k write. That's why we're adding the SSDs -- you just can't run a
proportioned RB
Hi Erwin,
Did you try and restart the primary osd for that pg (24) - sometimes it
needs a little ..nudge that way.
Otherwise what does ceph pg dump say about that pg?
Cheers,
Martin
On Thu, Sep 4, 2014 at 9:00 AM, Erwin Lubbers
wrote:
> Hi,
>
> My cluster is giving one stuck pg which seems t
Thanks again for all of your input. I agree with your assessment -- in our
cluster we avg 40ms for a 4k write. That's why we're adding the SSDs -- you
just can't run a proportioned RBD service without them.
I'll definitely give bcache a try in my test setup, but more reading has kinda
tempere
You should be able to use any block device in a bcache device. Right now,
we are OK losing one SSD and it takes out 5 OSDs. We would rather have
twice the cache. Our opinion may change in the future. We wanted to keep as
much overhead as low as possible. I think we may spend the extra on heavier
du
I've just been reading the bcache docs. It's a pity the mirrored writes aren't
implemented yet. Do you know if you can use an md RAID1 as a cache dev? And is
the graceful failover from wb to writethrough actually working without data
loss?
Also, write behind sure would help the filestore, since
So far it was worked really well, we can raise/lower/disable/enable the
cache in realtime and watch how the load and traffic changes. There has
been some positive subjective results, but definitive results are still
forth coming. bcache on CentOS 7 was not easy, makes me wish we were
running Debian
Hi,
I need more info on Ceph object backup mechanism.. Could someone share a
related doc or link for this?
Thanks
Swami
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Hi Robert,
That's actually a pretty good idea, since bcache would also accelerate the
filestore flushes and leveldb. I actually wonder if an SSD-only pool would even
be faster than such a setup... probably not.
We're using an ancient enterprise n distro, so it will be a bit of a headache
to ge
We are still pretty early on in our testing of how to best use SSDs as
well. What we are trying right now, for some of the reasons you mentioned
already, is to use bcache as a cache for both journal and data. We have 10
spindles in our boxes with 2 SSDs. We created two bcaches (one for each
SSD) an
Hey all,
The schedule for Ceph Day Paris has just been finalized and posted.
Looks like we have another great lineup of speakers (and some
never-before-seen topics) so if you are anywhere close to Paris on 18
Sep, you should definitely sign up soon!
http://ceph.com/cephdays/paris/
If you are a c
Dear Cephalopods,
In a few weeks we will receive a batch of 200GB Intel DC S3700’s to augment our
cluster, and I’d like to hear your practical experience and discuss options how
best to deploy these.
We’ll be able to equip each of our 24-disk OSD servers with 4 SSDs, so they
will become 20 OSD
Hello,
On Thu, 4 Sep 2014 20:56:31 +0800 Ding Dinghua wrote:
Aside from what Loic wrote, why not replace the network controller or if
it is onboard, add a card?
> Hi all,
> I'm new to ceph, and apologize if the question has been asked.
>
> I have setup a 8-nodes ceph cluster, a
Hi,
If the new machine can host the disks of the former machine, it should be
enough to
a) install the new machine with ceph
b) shutdown the old and new machines
c) move the disks from the old machine to the new
d) reboot the new machine
and the OSDs will show as if nothing happened.
Cheers
P
Hi all,
I'm new to ceph, and apologize if the question has been asked.
I have setup a 8-nodes ceph cluster, and after two months running,
network controller of an node is broken, so I have to replace the node with
an new one.
I don't want to trigger data migration, since al
Hello guys,
I was wondering if there is a benefit of using journal-less btrfs file system
on the cache pool osds? Would it speed up the writes to the cache tier? Is
btrfs and ceph getting close to production level?
Cheers
Andrei
___
ceph-users mailin
Vlad, thanks for the information. I will review it shortly.
I do have SSDs and SAS (not sata) disks in the same box. But I guess there
shouldn't be much difference between SAS and SATA.
At the moment I am running firefly. I've seen some comments that the master
branch has a great deal of improv
which version of MDS are you using?
On Wed, Sep 3, 2014 at 10:48 PM, Florent Bautista wrote:
> Hi John and thank you for your answer.
>
> I "solved" the problem doing : ceph mds stop 1
>
> So one MDS is marked as "stopping". A few hours later, it is still
> "stopping" (active process, consuming C
Hi,
My cluster is giving one stuck pg which seems to be backfilling for days now.
Any suggestions on how to solve it?
HEALTH_WARN 1 pgs backfilling; 1 pgs stuck unclean; recovery 32/6000626
degraded (0.001%)
pg 206.3f is stuck unclean for 557655.601540, current state
active+remapped+backfillin
54 matches
Mail list logo
