Is there a way to enable the LUKS encryption format on a snapshot that was
created from an unencrypted image without losing data? I've seen in
https://docs.ceph.com/en/quincy/rbd/rbd-encryption/ that "Any data written to
the image prior to its format may become unreadable, though it may still
Hi Dan,
Thanks for the response. No I have not yet told the OSDs participating in that
PG to compact. It was something I had thought about, but was somewhat concerned
about what that might do, or what performance impact that might have (or if the
OSD would come out alive on the other side). I
Tried [1] already, but got error.
Created no osd(s) on host ceph-4; already created?
The error is from [2] in deploy_osd_daemons_for_existing_osds().
Not sure what's missing.
Should OSD be removed, or removed with --replace, or untouched before host
reinstallation?
[1]
Hi,
The cluster is with Pacific and deployed by cephadm on container.
The case is to import OSDs after host OS reinstallation.
All OSDs are SSD who has DB/WAL and data together.
Did some research, but not able to find a working solution.
Wondering if anyone has experiences in this?
What needs to
There's a default/hard limit of 50 snaps that's maintained for any dir via
the definition MAX_SNAPS_PER_PATH = 50 in the source file
src/pybind/mgr/snap_schedule/fs/schedule_client.py.
Every time the snapshot names are read for pruning, the last thing done is
to check the length of the list and
On Fri, Apr 21, 2023 at 7:26 AM Kilian Ries wrote:
>
> Still didn't find out what will happen when the pool is full - but tried a
> little bit in our testing environment and i were not able to get the pool
> full before an OSD got full. So in first place one OSD reached the full ratio
> (pool
Thanks, Igor. I mentioned earlier that according to the OSD logs compaction
wasn't an issue. I did run `ceph-kvstore-tool` offline though, it completed
rather quickly without any warnings or errors, but the OSD kept showing
excessive latency.
I did something rather radical: rebooted the node and
Hi Angelo,
Just some thoughts to consider from our experience with similar setups:
1. Use Proxmox instead of VMWare, or anything KVM based. These VMs can
consume Ceph directly, and provide the same level of service (some may say
better) for live ,migration, hyperconvergence etc. Then you run
There is also a direct RBD client for MS Windows, though it's relatively young.
> On Apr 27, 2023, at 18:20, Bailey Allison wrote:
>
> Hey Angelo,
>
> Just to make sure I'm understanding correctly, the main idea for the use
> case is to be able to present Ceph storage to windows clients as
Hey Angelo,
Just to make sure I'm understanding correctly, the main idea for the use
case is to be able to present Ceph storage to windows clients as SMB?
If so, you can absolutely use CephFS to get that done. This is something we
do all the time with our cluster configurations, if we're
Details of this release are summarized here:
https://tracker.ceph.com/issues/59542#note-1
Release Notes - TBD
Seeking approvals for:
smoke - Radek, Laura
rados - Radek, Laura
rook - Sébastien Han
cephadm - Adam K
dashboard - Ernesto
rgw - Casey
rbd - Ilya
krbd - Ilya
fs - Venky, Patrick
Hey guys and girls,
I'm working on a project to build storage for one of our departments,
and I want to ask you guys and girls for input on the high-level
overview part. It's a long one, I hope you read along and comment.
SUMMARY
I made a plan last year to build a 'storage solution' including
Hi Zakhar,
you might want to try offline DB compaction using ceph-kvstore-tool for
this specific OSD.
Periodically we observe OSD perf drop due to degraded RocksDB
performance, particularly after bulk data removal/migration.. Compaction
is quite helpful in this case.
Thanks,
Igor
On
Hi, I asked a similar question about increasing scrub throughput some time ago
and couldn't get a fully satisfying answer:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/NHOHZLVQ3CKM7P7XJWGVXZUXY24ZE7RK
My observation is that much fewer (deep) scrubs are scheduled than could be
Luminous 12.2.11 is still default using bluefs_buffered_io = false, which will
disable the kernel side cache for rocksdb.
It's possible that your NVMe is saturated with the massive rocksdb workload
when bluefs_buffered_io is disabled.
So one way you could try is to set the bluefs_buffered_io =
On a 38 TB cluster, if you scrub 8 MB/s on 10 disks (using only numbers
already divided by replication factor), you need 55 days to scrub it once.
That's 8x larger than the default scrub factor [...] Also, even if I set
the default scrub interval to 8x larger, it my disks will still be thrashing
We have a large cluster on Quincy 17.2.3 with a bucket holding 8.9 million
small (15~20 MiB) objects.
All the objects were multipart uploads from scripts using `aws s3 cp`
The data is static (write-once, read-many) with no manual deletions and no new
writes for months.
We recently found 3
I don't think it's a commit from yesterday, I had this issue since last
week,
the command "ceph features" shows me that my clients have the luminous
versions, but I don't know how to upgrade client version (ceph osd
set-require-min-compat-client is not upgrading client version)
---
Nguetchouang
On Thu, Apr 27, 2023 at 11:36 AM Tarrago, Eli (RIS-BCT)
wrote:
>
> After working on this issue for a bit.
> The active plan is to fail over master, to the “west” dc. Perform a realm
> pull from the west so that it forces the failover to occur. Then have the
> “east” DC, then pull the realm data
After working on this issue for a bit.
The active plan is to fail over master, to the “west” dc. Perform a realm pull
from the west so that it forces the failover to occur. Then have the “east” DC,
then pull the realm data back. Hopefully will get both sides back in sync..
My concern with this
Hi,
I think the sasl handshake is the issue:
On gateway:
2023-04-26T15:25:49.341+0700 7f7f04d21700 1 ERROR: failed to create push
endpoint: due to: pubsub endpoint configuration error: unknown schema in:
2023-04-26T15:25:49.341+0700 7f7f04d21700 5 req 245249540 0.00365s
s3:delete_obj
Eugen,
Thanks again for your suggestions! The cluster is balanced, OSDs on this
host and other OSDs in the cluster are almost evenly utilized:
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL%USE VAR PGS STATUS
...
11hdd 9.38680 1.0 9.4 TiB 1.2
Ok, I was able to do a backflip and revert to the old index files:
# Get stuff
radosgw-admin metadata get bucket.instance:BUCKET_NAME:NEW_BUCKET_ID >
bucket.instance:BUCKET_NAME:NEW_BUCKET_ID.json
radosgw-admin metadata get bucket:BUCKET_NAME > bucket:BUCKET_NAME.json
# create copy for fast
Hi Istvan,
Looks like you are using user/password and SSL on the communication
channels between RGW and the Kafka broker.
Maybe the issue is around the certificate? could you please increase RGW
debug logs to 20 and see if there are any kafka related errors there?
Yuval
On Tue, Apr 25, 2023 at
Hi Thomas,
Thanks for the detailed info!
RGW lua scripting was never tested in a cephadm deployment :-(
Opened a tracker: https://tracker.ceph.com/issues/59574 to make sure this
would work out of the box.
Yuval
On Tue, Apr 25, 2023 at 10:25 PM Thomas Bennett wrote:
> Hi ceph users,
>
> I've
I don't see anything obvious in the pg output, they are relatively
small and don't hold many objects. If deep-scrubs would impact
performance that much you would see that in the iostat output as well.
Have you watched it for a while, maybe with -xmt options to see the
%util column as well?
Good morning,
After upgrading from Octopus (15.2.17) to Pacific (16.2.12) two days
ago, I'm noticing that the MGR daemons keep failing over to standby and
then back every 24hrs. Watching the output of 'ceph orch ps' I can see
that the memory consumption of the mgr is steadily growing until
> Indeed! Every Ceph instance I have seen (not many) and almost every HPC
> storage system I have seen have this problem, and that's because they were
> never setup to have enough IOPS to support the maintenance load, never mind
> the maintenance load plus the user load (and as a rule not even
To clarify a bit:
The bucket data is not in the main zonegroup.
I wanted to start the reshard in the zonegroup where the bucket and the
data is located, but rgw told me to do it in the primary zonegroup.
So I did it there and the index on the zonegroup where the bucket is
located is empty.
We
Hi,
I just resharded a bucket on an octopus multisite environment from 11 to
101.
I did it on the master zone and it went through very fast.
But now the index is empty.
The files are still there when doing a radosgw-admin bucket radoslist
--bucket-id
Do I just need to wait or do I need to
Hi,
First of all I would suggest upgrading your cluster on one of the supported
releases.
I think full recovery is recommended to get back the mds.
1. Stop the mdses and all the clients.
2. Fail the fs.
a. ceph fs fail
3. Backup the journal: (If the below command fails, make rados level
Cheers Dan,
would it be an option to enable the ops log? I still didn't figure out how
it is actually working.
But I am also thinking to move to the logparsing in HAproxy and disable the
access log on the RGW instances.
Am Mi., 26. Apr. 2023 um 18:21 Uhr schrieb Dan van der Ster <
No this is a cephadm setup. Not rook.
In the last days it is still deep scrubbing and filling up. We have to
do something about it as it now impacts our K8s cluster (very slow
cephfs access) and we are running out of (allocated) diskspace again.
Some more details now that I had a few more
Thanks Janne, I will hand that to the customer.
> Look at https://community.veeam.com/blogs-and-podcasts-57/sobr-veeam
> -capacity-tier-calculations-and-considerations-in-v11-2548
> for "extra large blocks" to make them 8M at least.
> We had one Veeam installation vomit millions of files onto our
> On a 38 TB cluster, if you scrub 8 MB/s on 10 disks (using only
> numbers already divided by replication factor), you need 55 days
> to scrub it once.
> That's 8x larger than the default scrub factor [...] Also, even
> if I set the default scrub interval to 8x larger, it my disks
> will still
> On a 38 TB cluster, if you scrub 8 MB/s on 10 disks (using only
> numbers already divided by replication factor), you need 55 days
> to scrub it once.
> That's 8x larger than the default scrub factor [...] Also, even
> if I set the default scrub interval to 8x larger, it my disks
> will still
Thanks, Eugen. I very much appreciate your time and replies.
It's a hybrid OSD with DB/WAL on NVME (Micron_7300_MTFDHBE1T6TDG) and block
storage on HDD (Toshiba MG06SCA10TE). There are 6 uniform hosts with 2 x
DB/WAL NVMEs and 9 x HDDs each, each NVME hosts DB/WAL for 4-5 OSDs. The
cluster was
Hi sur5r,
Am 4/27/23 um 10:33 schrieb Jakob Haufe:
> On Thu, 27 Apr 2023 09:07:10 +0200
> Tobias Hachmer wrote:
>
>> But we observed that max 50 snapshot are preserved. If a new snapshot is
>> created the oldest 51st is deleted.
>>
>> Is there a limit for maximum cephfs snapshots or maybe this
> >
> > > The question you should ask yourself, why you want to
> > change/investigate this?
> >
> > Because if scrubbing takes 10x longer thrashing seeks, my scrubs never
> > finish in time (the default is 1 week).
> > I end with e.g.
> >
> > > 267 pgs not deep-scrubbed in time
> >
> > On a 38
On Thu, 27 Apr 2023 09:07:10 +0200
Tobias Hachmer wrote:
> But we observed that max 50 snapshot are preserved. If a new snapshot is
> created the oldest 51st is deleted.
>
> Is there a limit for maximum cephfs snapshots or maybe this is a bug?
I've been wondering the same thing for about 6
>
> > The question you should ask yourself, why you want to
> change/investigate this?
>
> Because if scrubbing takes 10x longer thrashing seeks, my scrubs never
> finish in time (the default is 1 week).
> I end with e.g.
>
> > 267 pgs not deep-scrubbed in time
>
> On a 38 TB cluster, if you
Those numbers look really high to me, more than 2 seconds for a write
is awful. Is this a HDD-only cluster/pool? But even then it would be
too high, I just compared with our HDD-backed cluster (although
rocksDB is SSD-backed) which also mainly serves RBD to openstack. What
is the general
Thanks, Eugen!
It's a bunch of entries like this https://pastebin.com/TGPu6PAT - I'm not
really sure what to make of them. I checked adjacent OSDs and they have
similar ops, but aren't showing excessive latency.
/Z
On Thu, 27 Apr 2023 at 10:42, Eugen Block wrote:
> Hi,
>
> I would monitor the
Hi,
I would monitor the historic_ops_by_duration for a while and see if
any specific operation takes unusually long.
# this is within the container
[ceph: root@storage01 /]# ceph daemon osd.0 dump_historic_ops_by_duration
| head
{
"size": 20,
"duration": 600,
"ops": [
{
Hello,
we are running a 3-node ceph cluster with version 17.2.6.
For CephFS snapshots we have configured the following snap schedule with
retention:
/PATH 2h 72h15d6m
But we observed that max 50 snapshot are preserved. If a new snapshot is
created the oldest 51st is deleted.
Is there a
Den ons 26 apr. 2023 kl 21:20 skrev Niklas Hambüchen :
> > 100MB/s is sequential, your scrubbing is random. afaik everything is random.
>
> Is there any docs that explain this, any code, or other definitive answer?
> Also wouldn't it make sense that for scrubbing to be able to read the disk
>
46 matches
Mail list logo
