Hi,
I've recently upgraded from Nautilus 14.2.2 to 14.2.6. I've also been
installing some new OSDs to my cluster. It looks as though either the
backplane I've added has power issues or the raid card I've added has bad
memory. Several new-ish, known good drives were bounced out of their JBOD
Injectargs causes an immediate runtime change; rebooting the mon would
negate the change.
On Wed., Jan. 22, 2020, 4:41 p.m. St-Germain, Sylvain (SSC/SPC), <
sylvain.st-germ...@canada.ca> wrote:
> / Problem ///
>
> I've got a Warning on my cluster
/ Problem ///
I've got a Warning on my cluster that I cannot remove :
"1 pools have many more objects per pg than average"
Does somebody has some insight ? I think it's normal to have this warning
because I have just one pool in use, but how can
On Tue, Jan 21, 2020 at 05:57:51PM +, Robin H. Johnson wrote:
> On Mon, Jan 20, 2020 at 12:57:51PM +, EDH - Manuel Rios wrote:
> > Hi Cephs
> >
> > Several nodes of our Ceph 14.2.5 are fully dedicated to host cold storage /
> > backups information.
> >
> > Today checking the data usage
We have a CEPH storage cluster which is having problems.
When I run a rados bench I get the behaviour below. Has anyone seen this
sort of thing before?
# rados bench -p scbench 10 seq
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg
lat(s)
0 0 0
Thanks, Ilya.
I just tried modifying the osd cap for client.testuser by getting rid of
"tag cephfs data=cephfs_test" part and confirmed this key does work (i.e.
lets the CephFS client read/write). It now reads:
[client.testuser]
key = XXXZZZ
caps mds = "allow rw"
caps mon = "allow r"
caps
Hi Martin,
looks like a bug to me.
You might want to remove all custom settings from config database and
try to set osd-memory-target only.
Would it help?
Thanks,
Igor
On 1/22/2020 3:43 PM, Martin Mlynář wrote:
Dne 21. 01. 20 v 21:12 Stefan Kooman napsal(a):
Quoting Martin Mlynář
Hi,
Currently I have a cluster of 2 nodes on two raspberrypi 3 devices.
pi1 is admin/mon/mgr/osd
pi2 is osd
I am currently trying to run radosgw on pi2. I have failed multiple times
/var/lib/ceph/radosgw# /usr/bin/radosgw -f --cluster ceph --name
client.rgw.pi2 --setuser ceph --setgroup ceph
> Couldn't John confirm that this is the issue by checking the heap stats and
> triggering the release via
>
> ceph tell mds.mds1 heap stats
> ceph tell mds.mds1 heap release
>
> (this would be much less disruptive than restarting the MDS)
That was my first thought as well, but `release`
Dne 21. 01. 20 v 21:12 Stefan Kooman napsal(a):
> Quoting Martin Mlynář (nexus+c...@smoula.net):
>
>> Do you think this could help? OSD does not even start, I'm getting a little
>> lost how flushing caches could help.
> I might have mis-understood. I though the OSDs crashed when you set the
>
I don't find any clue from the backtrace. please run 'ceph daemon
mds. dump_historic_ops' and ''ceph daemon mds.xxx perf reset; ceph
daemon mds.xxx perf dump'. send the outputs to us.
Hi, I assume you mean ceph daemon mds.xxx perf reset _all_?
Here's the output of historic ops
On Wed, Jan 22, 2020 at 12:24 AM Patrick Donnelly
wrote:
> On Tue, Jan 21, 2020 at 8:32 AM John Madden wrote:
> >
> > On 14.2.5 but also present in Luminous, buffer_anon memory use spirals
> > out of control when scanning many thousands of files. The use case is
> > more or less "look up this
Hi,
We upgraded our Ceph cluster from Hammer to Luminous and it is running
fine. Post upgrade we live migrated all our Openstack instances (not 100%
sure). Currently we see 1658 clients still on Hammer version. To track the
clients we increased the debugging of debug_mon=10/10, debug_ms=1/5,
Hi Cbodley ,
As you requested by IRC we tested directly with AWS Cli.
Results:
aws --endpoint=http://XX --profile=ceph s3api list-multipart-uploads
--bucket Evol6
It reports near 170 uploads.
We used the last one:
{
"Initiator": {
"DisplayName": "x",
On Tue, Jan 21, 2020 at 8:32 AM John Madden wrote:
>
> On 14.2.5 but also present in Luminous, buffer_anon memory use spirals
> out of control when scanning many thousands of files. The use case is
> more or less "look up this file and if it exists append this chunk to
> it, otherwise create it
Hi Vitaliy,
The drive is a Micron 5200 ECO 3.84TB
This is from the msecli utility:
Device Name : /dev/sde
Model No : Micron_5200_MTFDDAK3T8TDC
Serial No:
FW-Rev : D1MU404
Total Size : 3840.00GB
Drive Status : Drive is in good
Hi! Thanks.
The parameter gets reset when you reconnect the SSD so in fact it requires not
to power cycle it after changing the parameter :-)
Ok, this case seems lucky, ~2x change isn't a lot. Can you tell the exact model
and capacity of this Micron, and what controller was used in this test?
We were able to isolate an individual Micron 5200 and perform Vitaliy's
tests in his spreadsheet.
An interesting item - write cache changes do NOT require a power cycle
to take effect, at least on a Micron 5200.
The complete results from fio are included at the end of this message
for the
Quoting Martin Mlynář (nexus+c...@smoula.net):
> Do you think this could help? OSD does not even start, I'm getting a little
> lost how flushing caches could help.
I might have mis-understood. I though the OSDs crashed when you set the
config setting.
> According to trace I suspect something
Dne út 21. 1. 2020 17:09 uživatel Stefan Kooman napsal:
> Quoting Martin Mlynář (nexus+c...@smoula.net):
>
> >
> > When I remove this option:
> > # ceph config rm osd osd_memory_target
> >
> > OSD starts without any trouble. I've seen same behaviour when I wrote
> > this parameter into
On Tue, Jan 21, 2020 at 7:51 PM Hayashida, Mami wrote:
>
> Ilya,
>
> Thank you for your suggestions!
>
> `dmsg` (on the client node) only had `libceph: mon0 10.33.70.222:6789 socket
> error on write`. No further detail. But using the admin key (client.admin)
> for mounting CephFS solved my
Hi Robin,
- What are the external tools? CloudBerry S3 Explorer and S3 Browser
- How many objects do the external tools report as existing? Tool report 72142
keys (Aprox 6TB) vs CEPH num_objects 180981 (9TB)
- Do the external tools include incomplete multipart uploads in their size
data? I
Ilya,
Thank you for your suggestions!
`dmsg` (on the client node) only had `libceph: mon0 10.33.70.222:6789
socket error on write`. No further detail. But using the admin key
(client.admin) for mounting CephFS solved my problem. I was able to write
successfully! :-)
$ sudo mount -t ceph
On Tue, Jan 21, 2020 at 6:02 PM Hayashida, Mami wrote:
>
> I am trying to set up a CephFS with a Cache Tier (for data) on a mini test
> cluster, but a kernel-mount CephFS client is unable to write. Cache tier
> setup alone seems to be working fine (I tested it with `rados put` and `osd
> map`
On Mon, Jan 20, 2020 at 12:57:51PM +, EDH - Manuel Rios wrote:
> Hi Cephs
>
> Several nodes of our Ceph 14.2.5 are fully dedicated to host cold storage /
> backups information.
>
> Today checking the data usage with a customer found that rgw-admin is
> reporting:
...
> That's near 5TB used
I am trying to set up a CephFS with a Cache Tier (for data) on a mini test
cluster, but a kernel-mount CephFS client is unable to write. Cache tier
setup alone seems to be working fine (I tested it with `rados put` and `osd
map` commands to verify on which OSDs the objects are placed) and setting
On 14.2.5 but also present in Luminous, buffer_anon memory use spirals
out of control when scanning many thousands of files. The use case is
more or less "look up this file and if it exists append this chunk to
it, otherwise create it with this chunk." The memory is recovered as
soon as the
Quoting Martin Mlynář (nexus+c...@smoula.net):
>
> When I remove this option:
> # ceph config rm osd osd_memory_target
>
> OSD starts without any trouble. I've seen same behaviour when I wrote
> this parameter into /etc/ceph/ceph.conf
>
> Is this a known bug? Am I doing something wrong?
I
Hi,
I'm having troubles changing osd_memory_target on my test cluster. I've
upgraded whole cluster from luminous to nautiuls, all OSDs are running
bluestore. Because this testlab is short in RAM, I wanted to lower
osd_memory_target to save some memory.
# ceph version
ceph version 14.2.6
OK, now I understand. Yes, the cache setting will take effect immediately. Its
more about do you trust the disk firmware to apply the change correctly in all
situations when production IO is active at the same time (will volatile cache
be flushed correctly or not)? I would not and rather change
Hello Igor,
thanks for all your feedback and all your help.
The first thing i'll try is to upgrade a bunch of system from
4.19.66 kernel to 4.19.97 and see what happens.
I'll report back in 7-10 days to verify whether this helps.
Greets,
Stefan
Am 20.01.20 um 13:12 schrieb Igor Fedotov:
> Hi
Frank,
Sorry for the confusion. I thought that turning off cache using hdparm -W
0 /dev/sdx takes effect right away and in case of non-raid controllers and
Seagate or Micron SSDs I would see a difference starting fio benchmark
right after executing hdparm. So I wonder it makes a difference
Hi everyone,
I'm trying to understand where is the difference between the command :
ceph df details
And the result I'm getting when I run this script :
total_bytes=0
while read user; do
echo $user
bytes=$(radosgw-admin user stats --uid=${user} | grep total_bytes_rounded |
tr -dc "0-9")
if
> So hdparam -W 0 /dev/sdx doesn't work or it makes no difference?
I wrote "We found the raw throughput in fio benchmarks to be very different for
write-cache enabled and disabled, exactly as explained in the performance
article.", so yes, it makes a huge difference.
> Also I am not sure I
On Tue, Jan 21, 2020 at 12:09 AM Janek Bevendorff
wrote:
>
> Hi, I did as you asked and created a thread dump with GDB on the
> blocking MDS. Here's the result: https://pastebin.com/pPbNvfdb
>
I don't find any clue from the backtrace. please run 'ceph daemon
mds. dump_historic_ops' and
Hi Vitaliy,
> You say you don't have access to raw drives. What does it mean? Do you
> run Ceph OSDs inside VMs? In that case you should probably disable
> Micron caches on the hosts, not just in VMs.
Sorry, I should have been more clear. This cluster is in production, so I
needed to schedule
It seems that people now split between new and old list servers.
Regardless of either one of them, I am missing a number of messages that
appear on archive pages but never seem to make to my inbox. And no they
are not in my junk folder. I wonder if some of my questions are not
getting a
Hi, I did as you asked and created a thread dump with GDB on the
blocking MDS. Here's the result: https://pastebin.com/pPbNvfdb
On 17/01/2020 13:07, Yan, Zheng wrote:
On Fri, Jan 17, 2020 at 4:47 PM Janek Bevendorff
wrote:
Hi,
We have a CephFS in our cluster with 3 MDS to which > 300
Yes, another MDS takes it over and even comes back, but client does not
always "unfreeze".
Weird, i see some different versions..
ceph versions
{
"mon": {
"ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4)
mimic (stable)": 2,
"ceph version 13.2.8
On 1/20/20 4:17 PM, Anton Aleksandrov wrote:
> Hello community,
>
> We have very small ceph cluster of just 12 OSDs (1 per small server), 3
> MDS (one active) and 1 cephFS client.
>
Which version of Ceph?
$ ceph versions
> CephFS client is running Centos7, kernel
Hello community,
We have very small ceph cluster of just 12 OSDs (1 per small server), 3
MDS (one active) and 1 cephFS client.
CephFS client is running Centos7, kernel 3.10.0-957.27.2.el7.x86_64.
We created 3 MDS servers for redundancy and we mount our filesystem by
connecting to 3 of them.
Hi Eric,
You say you don't have access to raw drives. What does it mean? Do you
run Ceph OSDs inside VMs? In that case you should probably disable
Micron caches on the hosts, not just in VMs.
Yes, disabling the write cache only takes place upon a power cycle... or
upon the next hotplug of
Hello,
I am fighting with rbd and CEPH_ARGS in order to make typing easier on a
client. First I created a keyring on one of the ceph nodes:
# ceph auth add client.rainer mon 'profile rbd' osd 'profile rbd'
added key for client.rainer
Then I added this keyring to /etc/ceph/ceph.keyring on a
So hdparam -W 0 /dev/sdx doesn't work or it makes no difference? Also I am
not sure I understand why it should happen before OSD have been started.
At least in my experience hdparam does it to hardware regardless.
On Mon, Jan 20, 2020, 2:25 AM Frank Schilder wrote:
> We are using Micron 5200
Hi Cephs
Several nodes of our Ceph 14.2.5 are fully dedicated to host cold storage /
backups information.
Today checking the data usage with a customer found that rgw-admin is reporting:
{
"bucket": "XX",
"tenant": "",
"zonegroup": "4d8c7c5f-ca40-4ee3-b5bb-b2cad90bd007",
Hi Stefan,
these lines are result of transaction dump performed on a failure during
transaction submission (which is shown as
"submit_transaction error: Corruption: block checksum mismatch code = 2"
Most probably they are out of interest (checksum errors are unlikely to
be caused by
No, bluestore_fsck_on_mount_deep is applied when bluestore_fsck_on_mount
is set to true only.
Hence there is no fsck on mount in your case.
Thanks,
Igor
On 1/20/2020 10:25 AM, huxia...@horebdata.cn wrote:
HI, Igor,
does this could cause the problem?
Den mån 20 jan. 2020 kl 09:03 skrev Dave Hall :
> Hello,
>
Since upgrading to Nautilus (+ Debian 10 Backports), when I issue
> 'ceph-volume lvm batch --bluestore ' it fails with
>
> bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
>
> I previously had Luminous + Debian 9
We are using Micron 5200 PRO, 1.92TB for RBD images on KVM and are very happy
with the performance. We are using EC 6+2 pools, which really eat up IOPs.
Still, we get enough performance out to run 20-50 VMs per disk, which results
in good space utilisation as well since our default image size
HI, Igor,
does this could cause the problem?
huxia...@horebdata.cn
From: Igor Fedotov
Date: 2020-01-19 11:41
To: huxia...@horebdata.cn; ceph-users
Subject: Re: [ceph-users] OSD up takes 15 minutes after machine restarts
Hi Samuel,
wondering if you have bluestore_fsck_on_mount option set to
Nigel,
Thanks. I've never seen that. Cool.
-Dave
Dave Hall
Binghamton University
On 1/19/2020 11:15 PM, Nigel Williams wrote:
On Mon, 20 Jan 2020 at 14:15, Dave Hall wrote:
BTW, I did try to search the list archives via
http://lists.ceph.com/pipermail/ceph-users-ceph.com/, but that
On Mon, 20 Jan 2020 at 14:15, Dave Hall wrote:
> BTW, I did try to search the list archives via
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/, but that didn't work
> well for me. Is there another way to search?
With your favorite search engine (say Goog / ddg ), you can do this:
Hello,
Since upgrading to Nautilus (+ Debian 10 Backports), when I issue
'ceph-volume lvm batch --bluestore ' it fails with
bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
I previously had Luminous + Debian 9 running on the same hardware with
the same OSD layout, but I
Hello Igor,
there's absolutely nothing in the logs before.
What do those lines mean:
Put( Prefix = O key =
0x7f8001cc45c881217262'd_data.4303206b8b4567.9632!='0xfffe6f0012'x'
Value size = 480)
Put( Prefix = O key =
Yes, except that this happens on 8 different clusters with different hw but
same ceph version and same kernel version.
Greets,
Stefan
> Am 19.01.2020 um 11:53 schrieb Igor Fedotov :
>
> So the intermediate summary is:
>
> Any OSD in the cluster can experience interim RocksDB checksum
So the intermediate summary is:
Any OSD in the cluster can experience interim RocksDB checksum failure.
Which isn't present after OSD restart.
No HW issues observed, no persistent artifacts (except OSD log) afterwards.
And looks like the issue is rather specific to the cluster as no similar
Hi Samuel,
wondering if you have bluestore_fsck_on_mount option set to true? Can
you see high read load over OSD device(s) during the startup?
If so it might be fsck running which takes that long.
Thanks,
Igor
On 1/19/2020 11:53 AM, huxia...@horebdata.cn wrote:
Dear folks,
I had a
Hi :
A server was just rebooted and the osd cant boot .The log is the following.
-3> 2020-01-19 17:39:25.904673 7f5b8e5e9d80 -1
bluestore(/var/lib/ceph/osd/ceph-44) _verify_csum bad crc32c/0x1000
checksum at blob offset 0x0, got 0xd2acc81f, expected 0x62cf539d,
device location
Dear folks,
I had a strange situation with 3-node Ceph cluster on Luminous 12.2.12 with
bluestore. Each machine has 5 OSDs on HDD, and each OSD uses a 30GB DB/WAL
partition on SSD. At the beginning without much data, OSDs can quickly up if
one node restarts.
Then I ran 4-day long stress
Hi Vitaliy,
Similar to Stefan, we have a bunch of Micron 5200's (3.84TB ECO SATA version)
in a Ceph cluster (Nautilus) and performance seems less than optimal. I have
followed all instructions on your site (thank you for your wonderful article
btw!!), but I haven't seen much change.
Gatherkeys and config push seem to have done the job. Thanks for your help,
Paul!
Justin
On Sat., Jan. 18, 2020, 02:33 Paul Emmerich, wrote:
> check if the mons have the same keyring file and the same config file.
> --
> Paul Emmerich
>
> Looking for help with your Ceph cluster? Contact us at
check if the mons have the same keyring file and the same config file.
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Sat, Jan 18, 2020 at 12:39 AM Justin Engwer wrote:
RGW tools will automatically deploy these pools, for example, running
radosgw-admin will create them if they don't exist.
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585
Hello,
I had very odd results in the past with the fio rbd engine and would
suggest testing things in the environment you're going to deploy in, end
to end.
That said, without any caching and coalescing of writes, sequential 4k
writes will hit the same set of OSDs for 4MB worth of data, thus
Hello,
I'm still a bit confused by the .rgw.root and the
default.rgw.{control,meta,log} pools.
I recently removed the RGW daemon I had running and the aforementioned
pools, however after a rebalance I suddenly find them again in the
output of:
$ ceph osd pool ls
cephfs_data
cephfs_metadata
Hi,
I'm a home user of ceph. Most of the time I can look at the email lists and
articles and figure things out on my own. I've unfortunately run into an
issue I can't troubleshoot myself.
Starting one of my monitors yields this error:
2020-01-17 15:34:13.497 7fca3d006040 0 mon.kvm2@-1(probing)
Not been able to make any headway on this after some significant effort.
-Tested all 48 SSDs with FIO directly, all tested with 10% of each other for 4k
iops in rand|seq read|write.
-Disabled all CPU power save.
-Tested with both rbd cache enabled and disabled on the client.
-Tested with drive
Frank,
Thank you for your input. It is good to know that the cluster will go
read-only in if a node goes down. Our circumstance is probably a bit
unusual, which is why I'm considering the2+1 solution. We have a
researcher who will be collecting extremely large amounts of data in
real
On Fri, 2020-01-17 at 17:10 +0100, Ilya Dryomov wrote:
> On Fri, Jan 17, 2020 at 2:21 AM Aaron wrote:
> > No worries, can definitely do that.
> >
> > Cheers
> > Aaron
> >
> > On Thu, Jan 16, 2020 at 8:08 PM Jeff Layton wrote:
> > > On Thu, 2020-01-16 at 18:42 -0500, Jeff Layton wrote:
> > > >
On Fri, Jan 17, 2020 at 2:21 AM Aaron wrote:
>
> No worries, can definitely do that.
>
> Cheers
> Aaron
>
> On Thu, Jan 16, 2020 at 8:08 PM Jeff Layton wrote:
>>
>> On Thu, 2020-01-16 at 18:42 -0500, Jeff Layton wrote:
>> > On Wed, 2020-01-15 at 08:05 -0500, Aaron wrote:
>> > > Seeing a weird
Actually, scratch that. I went ahead and opened this:
https://tracker.ceph.com/issues/43649
Feel free to watch that one for updates.
On Fri, 2020-01-17 at 07:43 -0500, Jeff Layton wrote:
> No problem. Can you let me know the tracker bug number once you've
> opened it?
>
> Thanks,
> Jeff
>
>
Thanks. I will do that. Right now, we have quite a few lags when listing
folders, which is probably due to another client heavily using the
system. Unfortunately, it's rather hard to debug at the moment, since
the suspected client has to use our Ganesha bridge instead of connecting
to the Ceph
On Fri, Jan 17, 2020 at 4:47 PM Janek Bevendorff
wrote:
>
> Hi,
>
> We have a CephFS in our cluster with 3 MDS to which > 300 clients
> connect at any given time. The FS contains about 80 TB of data and many
> million files, so it is important that meta data operations work
> smoothly even when
HI Igor,
Am 17.01.20 um 12:10 schrieb Igor Fedotov:
> hmmm..
>
> Just in case - suggest to check H/W errors with dmesg.
this happens on around 80 nodes - i don't expect all of those have not
identified hw errors. Also all of them are monitored - no dmesg outpout
contains any errors.
> Also
hmmm..
Just in case - suggest to check H/W errors with dmesg.
Also there are some (not very much though) chances this is another
incarnation of the following bug:
https://tracker.ceph.com/issues/22464
https://github.com/ceph/ceph/pull/24649
The corresponding PR works around it for main
I would strongly advise against 2+1 EC pools for production if stability is
your main concern. There was a discussion towards the end of last year
addressing this in more detail. Short story, if you don't have at least 8-10
nodes (in the short run), EC is not suitable. You cannot maintain a
Hi,
We have a CephFS in our cluster with 3 MDS to which > 300 clients
connect at any given time. The FS contains about 80 TB of data and many
million files, so it is important that meta data operations work
smoothly even when listing large directories.
Previously, we had massive stability
There is no difference in allocation between replication or EC. If failure
domain is host, one osd per host ok s used for a PG. So if you use a 2+1 EC
profile with a host failure domain, you need 3 hosts for a healthy cluster.
The pool will go read-only when you have a failure (host or disk), or
Bastiaan,
Regarding EC pools: Our concern at 3 nodes is that 2-way replication
seems risky - if the two copies don't match, which one is corrupted.
However, 3-way replication on a 3 node cluster triples the price per
TB. Doing EC pools that are the equivalent of RAID-5 2+1 seems like
This debugging started because the ceph-provisioner from k8s was making
those users...but what we found was doing something similar by hand caused
the same issue. Just surprised no one else using k8s and ceph backed
PVC/PVs ran into this issue.
Thanks again for all your help!
Cheers
Aaron
On
No worries, can definitely do that.
Cheers
Aaron
On Thu, Jan 16, 2020 at 8:08 PM Jeff Layton wrote:
> On Thu, 2020-01-16 at 18:42 -0500, Jeff Layton wrote:
> > On Wed, 2020-01-15 at 08:05 -0500, Aaron wrote:
> > > Seeing a weird mount issue. Some info:
> > >
> > > No LSB modules are
Paul;
So is the 3/30/300GB a limit of RocksDB, or of Bluestore?
The percentages you list, are they used DB / used data? If so... Where do you
get the used DB data from?
Thank you,
Dominic L. Hilsbos, MBA
Director – Information Technology
Perform Air International Inc.
Discussing DB size requirements without knowing the exact cluster
requirements doesn't work.
Here are some real-world examples:
cluster1: CephFS, mostly large files, replicated x3
0.2% used for metadata
cluster2: radosgw, mix between replicated and erasure, mixed file sizes
(lots of tiny files,
Hello,
We are trying to route backups & snapshots of cinder volume and nova instances,
into the s3 buckets hosted on ceph. Currently ceph is the block storage target
as well.
What we want to achieve ?
1. all snapshots of cinder volumes / nova instance to be routed to s3 buckets
of that
Dave made a good point WAL + DB might end up a little over 60G, I would
probably go with ~70Gig partitions /LV's per OSD in your case. (if the nvme
drive is smart enough to spread the writes over all available capacity,
mort recent nvme's are). I have not yet seen a WAL larger or even close to
Hello Stefan, but if I want to use rbd mirroring I must have site-a.conf
and site-b.conf on one of my nodes.probably one of the mon nodes. Is it
only a configuration on ceph client side ?
Thanks
Ignazio
Il Gio 16 Gen 2020, 22:13 Stefan Kooman ha scritto:
> Quoting Ignazio Cassano
Quoting Ignazio Cassano (ignaziocass...@gmail.com):
> Hello, I just deployed nautilus with ceph-deploy.
> I did not find any option to give a cluster name to my ceph so its name is
> "ceph".
> Please, how can I chenge my cluster name without reinstalling ?
>
> Please, how can I set the cluster
Hello, I just deployed nautilus with ceph-deploy.
I did not find any option to give a cluster name to my ceph so its name is
"ceph".
Please, how can I chenge my cluster name without reinstalling ?
Please, how can I set the cluster name in installation phase ?
Many thanks for help
Ignazio
Hi Igor,
answers inline.
Am 16.01.20 um 21:34 schrieb Igor Fedotov:
> you may want to run fsck against failing OSDs. Hopefully it will shed
> some light.
fsck just says everything fine:
# ceph-bluestore-tool --command fsck --path /var/lib/ceph/osd/ceph-27/
fsck success
> Also wondering if
Stefan,
you may want to run fsck against failing OSDs. Hopefully it will shed
some light.
Also wondering if OSD is able to recover (startup and proceed working)
after facing the issue?
If so do you have any one which failed multiple times? Do you have logs
for these occurrences?
Also
Dave;
I don't like reading inline responses, so...
I have zero experience with EC pools, so I won't pretend to give advice in that
area.
I would think that small NVMe for DB would be better than nothing, but I don't
know.
Once I got the hang of building clusters, it was relatively easy to
Dominic,
We ended up with a 1.6TB PCIe NVMe in each node. For 8 drives this
worked out to a DB size of something like 163GB per OSD. Allowing for
expansion to 12 drives brings it down to 124GB. So maybe just put the
WALs on NVMe and leave the DBs on the platters?
Understood that we will
Dave;
I'd like to expand on this answer, briefly...
The information in the docs is wrong. There have been many discussions about
changing it, but no good alternative has been suggested, thus it hasn't been
changed.
The 3rd party project that Ceph's BlueStore uses for its database (RocksDB),
Paul, Bastiaan,
Thank you for your responses and for alleviating my concerns about
Nautilus. The good news is that I can still easily move up to Debian
10. BTW, I assume that this is still with the 4.19 kernel?
Also, I'd like to inject additional customizations into my Debian
configs via
Don't use Mimic, support for it is far worse than Nautilus or Luminous. I
think we were the only company who built a product around Mimic, both
Redhat and Suse enterprise storage was Luminous and then Nautilus skipping
Mimic entirely.
We only offered Mimic as a default for a limited time and
Hi Igor,
ouch sorry. Here we go:
-1> 2020-01-16 01:10:13.404090 7f3350a14700 -1 rocksdb:
submit_transaction error: Corruption: block checksum mismatch code = 2
Rocksdb transaction:
Put( Prefix = M key =
I would definitely go for Nautilus. there are quite some optimizations that
went in after mimic.
Bluestore DB size usually ends up at either 30 or 60 GB.
30 GB is one of the sweet spots during normal operation. But during
compaction, ceph writes the new data before removing the old, hence the
Hello all.
Sorry for the beginner questions...
I am in the process of setting up a small (3 nodes, 288TB) Ceph cluster
to store some research data. It is expected that this cluster will grow
significantly in the next year, possibly to multiple petabytes and 10s
of nodes. At this time I'm
Hi Stefan,
would you please share log snippet prior the assertions? Looks like
RocksDB is failing during transaction submission...
Thanks,
Igor
On 1/16/2020 11:56 AM, Stefan Priebe - Profihost AG wrote:
Hello,
does anybody know a fix for this ASSERT / crash?
2020-01-16 02:02:31.316394
Hello,
We are trying to route backups & snapshots of cinder volume and nova instances,
into the s3 buckets hosted on ceph. Currently ceph is the block storage target
as well.
What we want to achieve ?
1. all snapshots of cinder volumes / nova instance to be routed to s3 buckets
of that
1 - 100 of 52714 matches
Mail list logo
