[ceph-users] Re: [RGW] radosgw does not respond after some time after upgrade from pacific to quincy

[Rok Jaklič]

It turned out that under heavy load one of the rgws have had too low soft
NOFILE limit set (max number of open files). We also accidentally found
that in our setup and hw the "round trips" under heavy load from frontend
proxy (nginx in our case) to radosgw cost is pretty high (let us say 50% in
capability of sent p/s), even if it is on the same machine and ip.

Thanks anyway.

On Wed, Jul 24, 2024 at 8:25 AM Eugen Block  wrote:

> Hi,
>
> can you tell a bit more about your setup? Are RGWs and OSDs colocated
> on the same servers? Are there any signs of a server overload like OOM
> killers or anything else related to the recovery? Are disks saturated?
> Is this cephadm managed? What's the current ceph status?
>
> Thanks,
> Eugen
>
> Zitat von Rok Jaklič :
>
> > Hi,
> >
> > we've just updated from pacific(16.2.15) to quincy(17.2.7) and everything
> > seems to work, however after some time radosgw stops responding and we
> have
> > to restart it.
> >
> > At first look, it seems that radosgw stops responding sometimes during
> > recovery.
> >
> > Does this maybe have to do something with mclock
> > https://docs.ceph.com/en/quincy/rados/configuration/mclock-config-ref/ ?
> >
> > ceph.conf looks something like this:
> >
> > 
> >
> > [global]
> > fsid = ...
> > mon initial members = x1,x2
> > mon host =
> > # public network = 192.168.0.0/24
> > auth cluster required = none
> > auth service required = none
> > auth client required = none
> > ms_mon_client_mode = crc
> >
> > osd journal size = 1024
> > osd pool default size = 3
> > osd pool default min size = 2
> > osd pool default pg num = 128
> > osd pool default pgp num = 128
> > osd crush chooseleaf type = 1
> >
> > [osd]
> > osd_scrub_begin_hour = 18
> > osd_scrub_end_hour = 6
> > osd_class_update_on_start = false
> > osd_scrub_during_recovery = false #scrub during recovery
> > osd_scrub_max_interval = 1209600
> > osd_deep_scrub_interval = 1209600
> > osd_max_scrubs = 3
> > osd_scrub_load_threshold = 1
> >
> > [client.radosgw.mon2]
> > host = mon2
> > # keyring = /etc/ceph/ceph.client.radosgw.keyring
> > log_file = /var/log/ceph/client.radosgw.mon2.log
> > rgw_dns_name = ...
> > rgw_frontends = "beast port="
> > rgw_max_put_param_size = 15728640
> > rgw_crypt_require_ssl = false
> > rgw_max_concurrent_requests = 2048
> > 
> >
> > We have nginx in front of rgw which upstream is set to
> client.radosgw.mon2
> > port .
> >
> > Kind regards,
> > Rok
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [RGW] radosgw does not respond after some time after upgrade from pacific to quincy

[Rok Jaklič]

What I only see in rgw logs in that "hang time" is something like:
2024-07-23T20:00:45.666+0200 7fc751496700  2 rgw data changes log:
RGWDataChangesLog::ChangesRenewThread: start
2024-07-23T20:00:57.072+0200 7fc740c75700 20 rgw notify: INFO: next queues
processing will happen at: Tue Jul 23 20:01:27 2024
2024-07-23T20:01:07.666+0200 7fc751496700  2 rgw data changes log:
RGWDataChangesLog::ChangesRenewThread: start
2024-07-23T20:01:27.534+0200 7fc740c75700 20 rgw notify: INFO: next queues
processing will happen at: Tue Jul 23 20:01:57 2024

On Tue, Jul 23, 2024 at 7:58 PM Rok Jaklič  wrote:

> Hi,
>
> we've just updated from pacific(16.2.15) to quincy(17.2.7) and everything
> seems to work, however after some time radosgw stops responding and we have
> to restart it.
>
> At first look, it seems that radosgw stops responding sometimes during
> recovery.
>
> Does this maybe have to do something with mclock
> https://docs.ceph.com/en/quincy/rados/configuration/mclock-config-ref/ ?
>
> ceph.conf looks something like this:
>
> 
>
> [global]
> fsid = ...
> mon initial members = x1,x2
> mon host =
> # public network = 192.168.0.0/24
> auth cluster required = none
> auth service required = none
> auth client required = none
> ms_mon_client_mode = crc
>
> osd journal size = 1024
> osd pool default size = 3
> osd pool default min size = 2
> osd pool default pg num = 128
> osd pool default pgp num = 128
> osd crush chooseleaf type = 1
>
> [osd]
> osd_scrub_begin_hour = 18
> osd_scrub_end_hour = 6
> osd_class_update_on_start = false
> osd_scrub_during_recovery = false #scrub during recovery
> osd_scrub_max_interval = 1209600
> osd_deep_scrub_interval = 1209600
> osd_max_scrubs = 3
> osd_scrub_load_threshold = 1
>
> [client.radosgw.mon2]
> host = mon2
> # keyring = /etc/ceph/ceph.client.radosgw.keyring
> log_file = /var/log/ceph/client.radosgw.mon2.log
> rgw_dns_name = ...
> rgw_frontends = "beast port="
> rgw_max_put_param_size = 15728640
> rgw_crypt_require_ssl = false
> rgw_max_concurrent_requests = 2048
> 
>
> We have nginx in front of rgw which upstream is set to client.radosgw.mon2
> port .
>
> Kind regards,
> Rok
>
>
>
>
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] [RGW] radosgw does not respond after some time after upgrade from pacific to quincy

[Rok Jaklič]

Hi,

we've just updated from pacific(16.2.15) to quincy(17.2.7) and everything
seems to work, however after some time radosgw stops responding and we have
to restart it.

At first look, it seems that radosgw stops responding sometimes during
recovery.

Does this maybe have to do something with mclock
https://docs.ceph.com/en/quincy/rados/configuration/mclock-config-ref/ ?

ceph.conf looks something like this:



[global]
fsid = ...
mon initial members = x1,x2
mon host =
# public network = 192.168.0.0/24
auth cluster required = none
auth service required = none
auth client required = none
ms_mon_client_mode = crc

osd journal size = 1024
osd pool default size = 3
osd pool default min size = 2
osd pool default pg num = 128
osd pool default pgp num = 128
osd crush chooseleaf type = 1

[osd]
osd_scrub_begin_hour = 18
osd_scrub_end_hour = 6
osd_class_update_on_start = false
osd_scrub_during_recovery = false #scrub during recovery
osd_scrub_max_interval = 1209600
osd_deep_scrub_interval = 1209600
osd_max_scrubs = 3
osd_scrub_load_threshold = 1

[client.radosgw.mon2]
host = mon2
# keyring = /etc/ceph/ceph.client.radosgw.keyring
log_file = /var/log/ceph/client.radosgw.mon2.log
rgw_dns_name = ...
rgw_frontends = "beast port="
rgw_max_put_param_size = 15728640
rgw_crypt_require_ssl = false
rgw_max_concurrent_requests = 2048


We have nginx in front of rgw which upstream is set to client.radosgw.mon2
port .

Kind regards,
Rok
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: What is the proper way to setup Rados Gateway (RGW) under Ceph?

[Rok Jaklič]

You don't have to. You can serve rgw on the front end directly.

You:
1. set certificate with smth like: rgw_frontends = " ...
ssl_certificate=/etc/pki/ceph/cert.pem". We are using nginx on front end to
act as a proxy and for some other stuff.
2. delete line with rgw_crypt_require_ssl

... you should be ready to go. :)

Rok

On Mon, Feb 12, 2024 at 6:43 PM Michael Worsham 
wrote:

> So, just so I am clear – in addition to the steps below, will I also need
> to also install NGINX or HAProxy on the server to act as the front end?
>
>
>
> -- M
>
>
>
> *From:* Rok Jaklič 
> *Sent:* Monday, February 12, 2024 12:30 PM
> *To:* Michael Worsham 
> *Cc:* ceph-users@ceph.io
> *Subject:* Re: [ceph-users] Re: What is the proper way to setup Rados
> Gateway (RGW) under Ceph?
>
>
>
> This is an external email. Please take care when clicking links or opening
> attachments. When in doubt, check with the Help Desk or Security.
>
>
>
> Hi,
>
>
>
> recommended methods of deploying rgw are imho overly complicated. You can
> get service up manually also with something simple like:
>
>
>
> [root@mon1 bin]# cat /etc/ceph/ceph.conf
>
> [global]
> fsid = 12345678-XXXx ...
> mon initial members = mon1,mon3
> mon host = ip-mon1,ip-mon2
> auth cluster required = none
> auth service required = none
> auth client required = none
> ms_mon_client_mode = crc
>
> [client.radosgw.mon1]
> host = mon1
> log_file = /var/log/ceph/client.radosgw.mon1.log
> rgw_dns_name = mon1
> rgw_frontends = "civetweb port=80 num_threads=500" # this is different in
> ceph versions 17, 18.
> rgw_crypt_require_ssl = false
>
> 
>
> [root@mon1 bin]# cat start-rgw.sh
> radosgw -c /etc/ceph/ceph.conf --setuser ceph --setgroup ceph -n
> client.radosgw.mon1 &
>
>
>
> ---
>
>
>
> This configuration has nginx in front of rgw  all traffic goes from
> nginx 443 -> rgw 80 and it assumes you "own the network" and you are aware
> of "drawbacks".
>
>
>
> Rok
>
>
>
> On Mon, Feb 12, 2024 at 2:15 PM Michael Worsham <
> mwors...@datadimensions.com> wrote:
>
> Can anyone help me on this? I can't be that hard to do.
>
> -- Michael
>
>
> -Original Message-
> From: Michael Worsham 
> Sent: Thursday, February 8, 2024 3:03 PM
> To: ceph-users@ceph.io
> Subject: [ceph-users] What is the proper way to setup Rados Gateway (RGW)
> under Ceph?
>
> I have setup a 'reef' Ceph Cluster using Cephadm and Ansible in a VMware
> ESXi 7 / Ubuntu 22.04 lab environment per the how-to guide provided here:
> https://computingforgeeks.com/install-ceph-storage-cluster-on-ubuntu-linux-servers/
> .
>
> The installation steps were fairly easy and I was able to get the
> environment up and running in about 15 minutes under VMware ESXi 7. I have
> buckets and pools already setup. However, the ceph.io site is confusing
> on how to setup the Rados Gateway (radosgw) with Multi-site --
> https://docs.ceph.com/en/latest/radosgw/multisite/. Is a copy of HAProxy
> also needed for handling the front-end load balancing or is it implied that
> Ceph sets it up?
>
> Command-line scripting I was planning on using for setting up the RGW:
> ```
> radosgw-admin realm create --rgw-realm=sandbox --default radosgw-admin
> zonegroup create --rgw-zonegroup=sandbox  --master --default radosgw-admin
> zone create --rgw-zonegroup=sandbox --rgw-zone=sandbox --master --default
> radosgw-admin period update --rgw-realm=sandbox --commit ceph orch apply
> rgw sandbox --realm=sandbox --zone=sandbox --placement="2 ceph-mon1
> ceph-mon2" --port=8000 ```
>
> What other steps are needed to get the RGW up and running so that it can
> be presented to something like Veeam for doing performance and I/O testing
> concepts?
>
> -- Michael
>
> This message and its attachments are from Data Dimensions and are intended
> only for the use of the individual or entity to which it is addressed, and
> may contain information that is privileged, confidential, and exempt from
> disclosure under applicable law. If the reader of this message is not the
> intended recipient, or the employee or agent responsible for delivering the
> message to the intended recipient, you are hereby notified that any
> dissemination, distribution, or copying of this communication is strictly
> prohibited. If you have received this communication in error, please notify
> the sender immediately and permanently delete the original email and
> destroy any copies or printouts of this email as well as any attachments.
> ___
> ceph-users mailing list -- ceph-users@ce

[ceph-users] Re: What is the proper way to setup Rados Gateway (RGW) under Ceph?

[Rok Jaklič]

Hi,

recommended methods of deploying rgw are imho overly complicated. You can
get service up manually also with something simple like:

[root@mon1 bin]# cat /etc/ceph/ceph.conf

[global]
fsid = 12345678-XXXx ...
mon initial members = mon1,mon3
mon host = ip-mon1,ip-mon2
auth cluster required = none
auth service required = none
auth client required = none
ms_mon_client_mode = crc

[client.radosgw.mon1]
host = mon1
log_file = /var/log/ceph/client.radosgw.mon1.log
rgw_dns_name = mon1
rgw_frontends = "civetweb port=80 num_threads=500" # this is different in
ceph versions 17, 18.
rgw_crypt_require_ssl = false



[root@mon1 bin]# cat start-rgw.sh
radosgw -c /etc/ceph/ceph.conf --setuser ceph --setgroup ceph -n
client.radosgw.mon1 &

---

This configuration has nginx in front of rgw  all traffic goes from
nginx 443 -> rgw 80 and it assumes you "own the network" and you are aware
of "drawbacks".

Rok

On Mon, Feb 12, 2024 at 2:15 PM Michael Worsham 
wrote:

> Can anyone help me on this? I can't be that hard to do.
>
> -- Michael
>
>
> -Original Message-
> From: Michael Worsham 
> Sent: Thursday, February 8, 2024 3:03 PM
> To: ceph-users@ceph.io
> Subject: [ceph-users] What is the proper way to setup Rados Gateway (RGW)
> under Ceph?
>
> I have setup a 'reef' Ceph Cluster using Cephadm and Ansible in a VMware
> ESXi 7 / Ubuntu 22.04 lab environment per the how-to guide provided here:
> https://computingforgeeks.com/install-ceph-storage-cluster-on-ubuntu-linux-servers/
> .
>
> The installation steps were fairly easy and I was able to get the
> environment up and running in about 15 minutes under VMware ESXi 7. I have
> buckets and pools already setup. However, the ceph.io site is confusing
> on how to setup the Rados Gateway (radosgw) with Multi-site --
> https://docs.ceph.com/en/latest/radosgw/multisite/. Is a copy of HAProxy
> also needed for handling the front-end load balancing or is it implied that
> Ceph sets it up?
>
> Command-line scripting I was planning on using for setting up the RGW:
> ```
> radosgw-admin realm create --rgw-realm=sandbox --default radosgw-admin
> zonegroup create --rgw-zonegroup=sandbox  --master --default radosgw-admin
> zone create --rgw-zonegroup=sandbox --rgw-zone=sandbox --master --default
> radosgw-admin period update --rgw-realm=sandbox --commit ceph orch apply
> rgw sandbox --realm=sandbox --zone=sandbox --placement="2 ceph-mon1
> ceph-mon2" --port=8000 ```
>
> What other steps are needed to get the RGW up and running so that it can
> be presented to something like Veeam for doing performance and I/O testing
> concepts?
>
> -- Michael
>
> This message and its attachments are from Data Dimensions and are intended
> only for the use of the individual or entity to which it is addressed, and
> may contain information that is privileged, confidential, and exempt from
> disclosure under applicable law. If the reader of this message is not the
> intended recipient, or the employee or agent responsible for delivering the
> message to the intended recipient, you are hereby notified that any
> dissemination, distribution, or copying of this communication is strictly
> prohibited. If you have received this communication in error, please notify
> the sender immediately and permanently delete the original email and
> destroy any copies or printouts of this email as well as any attachments.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Etag change of a parent object

[Rok Jaklič]

Hi,

shouldn't etag of a "parent" object change when "child" objects are added
on s3?

Example:
1. I add an object to test bucket: "example/" - size 0
"example/" has an etag XYZ1
2. I add an object to test bucket: "example/test1.txt" - size 12
"example/test1.txt" has an etag XYZ2
"example/" has an etag XYZ1 ... should this change?

I understand that object storage is not hierarchical by design and objects
are "not connected" by some other means than the bucket name.

Kind regards,
Rok
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Uploading file from admin to other users bucket in multi tenant mode

[Rok Jaklič]

Hi,

I have set following permission to admin user:
radosgw-admin caps add --uid=admin --tenant=admin --caps="users=*;buckets=*"

Now I would like to upload some object with admin user to some other
user/tenant (tester1$tester1) to his bucket test1.

Other user has uid tester1 and tenant tester1 and bucket test1 created.

I've tried with python:
headers = {'x-amz-meta-tenancy': 'tester1'}
client.upload_file(file_path, bucket_name, object_name,
ExtraArgs={'Metadata': headers})

But I get response:
An error occurred (NoSuchBucket) when calling the PutObject operation: None

Any ideas why I get this error even though bucket test1 for tester1$tester1
exists?

Kind regards,
Rok
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Bucket/object create/update/delete notification

[Rok Jaklič]

Thx. Got it done with these two python scripts based on examples from your
provided link.

Rok


On Wed, Nov 29, 2023 at 11:13 AM Yuval Lifshitz  wrote:

> Hi Rok,
> Please have a look here:
> https://github.com/ceph/ceph/tree/main/examples/rgw/boto3
> It has both CLI and python examples. Let me know if you need more
> information.
>
> Yuval
>
> On Tue, Nov 28, 2023 at 10:21 PM Rok Jaklič  wrote:
>
>> Hi,
>>
>> I would like to get info if the bucket or object got updated.
>>
>> I can get this info with a changed etag of an object, but not I cannot get
>> etag from bucket, so I am looking at
>> https://docs.ceph.com/en/latest/radosgw/notifications/
>>
>> How do I create a topic and where do I send request with parameters?
>>
>> Anyone have an example (cli, python, ...) with topic creation where I can
>> get notification to a http endpoint when an object got created in a
>> bucket?
>>
>> Kind regards,
>> Rok
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Bucket/object create/update/delete notification

[Rok Jaklič]

Hi,

I would like to get info if the bucket or object got updated.

I can get this info with a changed etag of an object, but not I cannot get
etag from bucket, so I am looking at
https://docs.ceph.com/en/latest/radosgw/notifications/

How do I create a topic and where do I send request with parameters?

Anyone have an example (cli, python, ...) with topic creation where I can
get notification to a http endpoint when an object got created in a bucket?

Kind regards,
Rok
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Received signal: Hangup from killall

[Rok Jaklič]

After looking through documentation soft log kills are "normal", however in
radosgw logs we found:
2023-10-06T01:31:32.920+0200 7fb6f440b700  0 INFO: RGWReshardLock::lock
found lock on reshard.02 to be held by another RGW process;
skipping for now
2023-10-06T01:31:33.371+0200 7fb6f440b700  0 INFO: RGWReshardLock::lock
found lock on reshard.04 to be held by another RGW process;
skipping for now
2023-10-06T01:31:33.521+0200 7fb6f440b700  0 INFO: RGWReshardLock::lock
found lock on reshard.06 to be held by another RGW process;
skipping for now
2023-10-06T01:31:33.853+0200 7fb6f440b700  0 INFO: RGWReshardLock::lock
found lock on reshard.08 to be held by another RGW process;
skipping for now
2023-10-06T01:31:34.598+0200 7fb6f440b700  0 INFO: RGWReshardLock::lock
found lock on reshard.12 to be held by another RGW process;
skipping for now
2023-10-06T01:31:34.740+0200 7fb6f440b700  0 INFO: RGWReshardLock::lock
found lock on reshard.14 to be held by another RGW process;
skipping for now
...
after this line ... it seems that rgw stopped responding.

And the next day it stopped again almost at the same time
2023-10-07T01:27:26.299+0200 7f6216651700  0 INFO: RGWReshardLock::lock
found lock on reshard.05 to be held by another RGW process;
skipping for now
2023-10-07T01:37:28.077+0200 7f6216651700  0 INFO: RGWReshardLock::lock
found lock on reshard.14 to be held by another RGW process;
skipping for now
2023-10-07T01:47:27.333+0200 7f6216651700  0 INFO: RGWReshardLock::lock
found lock on reshard.01 to be held by another RGW process;
skipping for now
2023-10-07T02:47:29.863+0200 7f6216651700  0 INFO: RGWReshardLock::lock
found lock on reshard.06 to be held by another RGW process;
skipping for now
...
after this line ... rgw stopped responding. We had to restart it.

We were just about to upgrade to ceph 17.x... but we had postpone it
because of this.

Rok




On Fri, Oct 6, 2023 at 9:30 AM Rok Jaklič  wrote:

> Hi,
>
> yesterday we changed RGW from civetweb to beast and at 04:02 RGW stopped
> working; we had to restart it in the morning.
>
> In one rgw log for previous day we can see:
> 2023-10-06T04:02:01.105+0200 7fb71d45d700 -1 received  signal: Hangup from
> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw
> rbd-mirror cephfs-mirror  (PID: 3202663) UID: 0
> and in the next day log we can see:
> 2023-10-06T04:02:01.133+0200 7fb71d45d700 -1 received  signal: Hangup from
>  (PID: 3202664) UID: 0
>
> and after that no requests came. We had to restart rgw.
>
> In ceph.conf we have something like
>
> [client.radosgw.ctplmon2]
> host = ctplmon2
> log_file = /var/log/ceph/client.radosgw.ctplmon2.log
> rgw_dns_name = ctplmon2
> rgw_frontends = "beast ssl_endpoint=0.0.0.0:4443 ssl_certificate=..."
> rgw_max_put_param_size = 15728640
>
> We assume it has something to do with logrotate.
>
> /etc/logrotate.d/ceph:
> /var/log/ceph/*.log {
> rotate 90
> daily
> compress
> sharedscripts
> postrotate
> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse
> radosgw rbd-mirror cephfs-mirror || pkill -1 -x
> "ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph-fuse|radosgw|rbd-mirror|cephfs-mirror"
> || true
> endscript
> missingok
> notifempty
> su root ceph
> }
>
> ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific
> (stable)
>
> And ideas why this happend?
>
> Kind regards,
> Rok
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Received signal: Hangup from killall

[Rok Jaklič]

Hi,

yesterday we changed RGW from civetweb to beast and at 04:02 RGW stopped
working; we had to restart it in the morning.

In one rgw log for previous day we can see:
2023-10-06T04:02:01.105+0200 7fb71d45d700 -1 received  signal: Hangup from
killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw
rbd-mirror cephfs-mirror  (PID: 3202663) UID: 0
and in the next day log we can see:
2023-10-06T04:02:01.133+0200 7fb71d45d700 -1 received  signal: Hangup from
 (PID: 3202664) UID: 0

and after that no requests came. We had to restart rgw.

In ceph.conf we have something like

[client.radosgw.ctplmon2]
host = ctplmon2
log_file = /var/log/ceph/client.radosgw.ctplmon2.log
rgw_dns_name = ctplmon2
rgw_frontends = "beast ssl_endpoint=0.0.0.0:4443 ssl_certificate=..."
rgw_max_put_param_size = 15728640

We assume it has something to do with logrotate.

/etc/logrotate.d/ceph:
/var/log/ceph/*.log {
rotate 90
daily
compress
sharedscripts
postrotate
killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw
rbd-mirror cephfs-mirror || pkill -1 -x
"ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph-fuse|radosgw|rbd-mirror|cephfs-mirror"
|| true
endscript
missingok
notifempty
su root ceph
}

ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific
(stable)

And ideas why this happend?

Kind regards,
Rok
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph_leadership_team_meeting_s18e06.mkv

[Rok Jaklič]

I can confirm this. ... as we did the upgrade from .10 also.

Rok


On Fri, Sep 8, 2023 at 5:26 PM David Orman  wrote:

> I would suggest updating: https://tracker.ceph.com/issues/59580
>
> We did notice it with 16.2.13, as well, after upgrading from .10, so
> likely in-between those two releases.
>
> David
>
> On Fri, Sep 8, 2023, at 04:00, Loïc Tortay wrote:
> > On 07/09/2023 21:33, Mark Nelson wrote:
> >> Hi Rok,
> >>
> >> We're still try to catch what's causing the memory growth, so it's hard
> >> to guess at which releases are affected.  We know it's happening
> >> intermittently on a live Pacific cluster at least.  If you have the
> >> ability to catch it while it's happening, there are several
> >> approaches/tools that might aid in diagnosing it. Container deployments
> >> are a bit tougher to get debugging tools working in though which afaik
> >> has slowed down existing attempts at diagnosing the issue.
> >>
> > Hello,
> > We have a cluster recently upgraded from Octopus to Pacific 16.2.13
> > where the active MGR was OOM-killed a few times.
> >
> > We have another cluster that was recently upgraded from 16.2.11 to
> > 16.2.14 and the issue also started to appear (very soon) on that cluster.
> > We didn't have the issue before, during the months running 16.2.11.
> >
> > In short: the issue seems to be due to a change in 16.2.12 or 16.2.13.
> >
> >
> > Loïc.
> > --
> > |   Loīc Tortay  - IN2P3 Computing Centre  |
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph_leadership_team_meeting_s18e06.mkv

[Rok Jaklič]

We do not use containers.

Anything special for debugging or should we try something from previous
email?
   - Enable profiling (Mark Nelson)
   - Try Bloomberg's Python mem profiler
   <https://github.com/bloomberg/memray> (Matthew Leonard)

Profiling means instructions from
https://docs.ceph.com/en/pacific/rados/troubleshooting/memory-profiling/ ?

Rok

On Thu, Sep 7, 2023 at 9:34 PM Mark Nelson  wrote:

> Hi Rok,
>
> We're still try to catch what's causing the memory growth, so it's hard
> to guess at which releases are affected.  We know it's happening
> intermittently on a live Pacific cluster at least.  If you have the
> ability to catch it while it's happening, there are several
> approaches/tools that might aid in diagnosing it. Container deployments
> are a bit tougher to get debugging tools working in though which afaik
> has slowed down existing attempts at diagnosing the issue.
>
> Mark
>
> On 9/7/23 05:55, Rok Jaklič wrote:
> > Hi,
> >
> > we have also experienced several ceph-mgr oom kills on ceph v16.2.13 on
> > 120T/200T data.
> >
> > Is there any tracker about the problem?
> >
> > Does upgrade to 17.x "solves" the problem?
> >
> > Kind regards,
> > Rok
> >
> >
> >
> > On Wed, Sep 6, 2023 at 9:36 PM Ernesto Puerta 
> wrote:
> >
> >> Dear Cephers,
> >>
> >> Today brought us an eventful CTL meeting: it looks like Jitsi recently
> >> started
> >> requiring user authentication
> >> <https://jitsi.org/blog/authentication-on-meet-jit-si/> (anonymous
> users
> >> will get a "Waiting for a moderator" modal), but authentication didn't
> work
> >> against Google or GitHub accounts, so we had to move to the good old
> Google
> >> Meet.
> >>
> >> As a result of this, Neha has kindly set up a new private Slack channel
> >> (#clt) to allow for quicker communication among CLT members (if you
> usually
> >> attend the CLT meeting and have not been added, please ping any CLT
> member
> >> to request that).
> >>
> >> Now, let's move on the important stuff:
> >>
> >> *The latest Pacific Release (v16.2.14)*
> >>
> >> *The Bad*
> >> The 14th drop of the Pacific release has landed with a few hiccups:
> >>
> >> - Some .deb packages were made available to downloads.ceph.com
> before
> >> the release process completion. Although this is not the first time
> it
> >> happens, we want to ensure this is the last one, so we'd like to
> gather
> >> ideas to improve the release publishing process. Neha encouraged
> >> everyone
> >> to share ideas here:
> >>- https://tracker.ceph.com/issues/62671
> >>- https://tracker.ceph.com/issues/62672
> >>- v16.2.14 also hit issues during the ceph-container stage. Laura
> >> wanted to raise awareness of its current setbacks
> >> <https://pad.ceph.com/p/16.2.14-struggles> and collect ideas to
> tackle
> >> them:
> >>- Enforce reviews and mandatory CI checks
> >>- Rework the current approach to use simple Dockerfiles
> >><https://github.com/ceph/ceph/pull/43292>
> >>- Call the Ceph community for help: ceph-container is currently
> >>maintained part-time by a single contributor (Guillaume Abrioux).
> >> This
> >>sub-project would benefit from the sound expertise on containers
> >> among Ceph
> >>users. If you have ever considered contributing to Ceph, but
> felt a
> >> bit
> >>intimidated by C++, Paxos and race conditions, ceph-container is
> a
> >> good
> >>place to shed your fear.
> >>
> >>
> >> *The Good*
> >> Not everything about v16.2.14 was going to be bleak: David Orman
> brought us
> >> really good news. They tested v16.2.14 on a large production cluster
> >> (10gbit/s+ RGW and ~13PiB raw) and found that it solved a major issue
> >> affecting RGW in Pacific <https://github.com/ceph/ceph/pull/52552>.
> >>
> >> *The Ugly*
> >> During that testing, they noticed that ceph-mgr was occasionally OOM
> killed
> >> (nothing new to 16.2.14, as it was previously reported). They already
> >> tried:
> >>
> >> - Disabling modules (like the restful one, which was a suspect)
> >> - Enabling debug 20
> >> - Turning the pg autoscale

[ceph-users] Re: ceph_leadership_team_meeting_s18e06.mkv

[Rok Jaklič]

Hi,

we have also experienced several ceph-mgr oom kills on ceph v16.2.13 on
120T/200T data.

Is there any tracker about the problem?

Does upgrade to 17.x "solves" the problem?

Kind regards,
Rok



On Wed, Sep 6, 2023 at 9:36 PM Ernesto Puerta  wrote:

> Dear Cephers,
>
> Today brought us an eventful CTL meeting: it looks like Jitsi recently
> started
> requiring user authentication
>  (anonymous users
> will get a "Waiting for a moderator" modal), but authentication didn't work
> against Google or GitHub accounts, so we had to move to the good old Google
> Meet.
>
> As a result of this, Neha has kindly set up a new private Slack channel
> (#clt) to allow for quicker communication among CLT members (if you usually
> attend the CLT meeting and have not been added, please ping any CLT member
> to request that).
>
> Now, let's move on the important stuff:
>
> *The latest Pacific Release (v16.2.14)*
>
> *The Bad*
> The 14th drop of the Pacific release has landed with a few hiccups:
>
>- Some .deb packages were made available to downloads.ceph.com before
>the release process completion. Although this is not the first time it
>happens, we want to ensure this is the last one, so we'd like to gather
>ideas to improve the release publishing process. Neha encouraged
> everyone
>to share ideas here:
>   - https://tracker.ceph.com/issues/62671
>   - https://tracker.ceph.com/issues/62672
>   - v16.2.14 also hit issues during the ceph-container stage. Laura
>wanted to raise awareness of its current setbacks
> and collect ideas to tackle
>them:
>   - Enforce reviews and mandatory CI checks
>   - Rework the current approach to use simple Dockerfiles
>   
>   - Call the Ceph community for help: ceph-container is currently
>   maintained part-time by a single contributor (Guillaume Abrioux).
> This
>   sub-project would benefit from the sound expertise on containers
> among Ceph
>   users. If you have ever considered contributing to Ceph, but felt a
> bit
>   intimidated by C++, Paxos and race conditions, ceph-container is a
> good
>   place to shed your fear.
>
>
> *The Good*
> Not everything about v16.2.14 was going to be bleak: David Orman brought us
> really good news. They tested v16.2.14 on a large production cluster
> (10gbit/s+ RGW and ~13PiB raw) and found that it solved a major issue
> affecting RGW in Pacific .
>
> *The Ugly*
> During that testing, they noticed that ceph-mgr was occasionally OOM killed
> (nothing new to 16.2.14, as it was previously reported). They already
> tried:
>
>- Disabling modules (like the restful one, which was a suspect)
>- Enabling debug 20
>- Turning the pg autoscaler off
>
> Debugging will continue to characterize this issue:
>
>- Enable profiling (Mark Nelson)
>- Try Bloomberg's Python mem profiler
> (Matthew Leonard)
>
>
> *Infrastructure*
>
> *Reminder: Infrastructure Meeting Tomorrow. **11:30-12:30 Central Time*
>
> Patrick brought up the following topics:
>
>- Need to reduce the OVH spending ($72k/year, which is a good cut in the
>Ceph Foundation budget, that's a lot less avocado sandwiches for the
> next
>Cephalocon):
>   - Move services (e.g.: Chacra) to the Sepia lab
>   - Re-use CentOS (and any spared/unused) machines for devel purposes
>- Current Ceph sys admins are overloaded, so devel/community involvement
>would be much appreciated.
>- More to be discussed in tomorrow's meeting. Please join if you
>think you can help solve/improve the Ceph infrastrucru!
>
>
> *BTW*: today's CDM will be canceled, since no topics were proposed.
>
> Kind Regards,
>
> Ernesto
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Applying crush rule to existing live pool

[Rok Jaklič]

Hi,

I want to place an existing pool with data to ssd-s.

I've created crush rule:
ceph osd crush rule create-replicated replicated_ssd default host ssd

If I apply this rule to the existing pool default.rgw.buckets.index with
180G of data with command:
ceph osd pool set default.rgw.buckets.index crush_rule replicated_ssd

Will the rgw/cluster be available in the meantime while the cluster moves
data?

Kind regards,
Rok
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: radosgw hang under pressure

[Rok Jaklič]

From
https://swamireddy.wordpress.com/2019/10/23/ceph-sharding-the-rados-gateway-bucket-index/

*Since the index is stored in a single RADOS object, only a single
operation can be done on it at any given time. When the number of objects
increases, the index stored in the RADOS object grows. Since a single index
is handling a large number of objects, and there is a chance the number of
operations also increases, parallelism is not possible which can end up
being a bottleneck. Multiple operations will need to wait in a queue since
a single operation is possible at a time.*

Maybe you know, is this still the case? Article is from 2019.




On Sun, Jun 25, 2023 at 6:22 PM Szabo, Istvan (Agoda) <
istvan.sz...@agoda.com> wrote:

> Hi,
>
> Can you check the read and write latency of your osds?
> Maybe it hangs because it’s waiting for pg’s but maybe the pg are under
> scrub or something else.
> Also with many small objects don’t rely on pg autoscaler, it might not
> tell to increase pg but maybe it should be.
>
> Istvan Szabo
> Staff Infrastructure Engineer
> ---
> Agoda Services Co., Ltd.
> e: istvan.sz...@agoda.com
> -------
>
> On 2023. Jun 23., at 19:12, Rok Jaklič  wrote:
>
> Email received from the internet. If in doubt, don't click any link nor
> open any attachment !
> 
>
> We are experiencing something similar (slow GETs responses) when sending 1k
> delete requests for example in ceph v16.2.13.
>
> Rok
>
> On Mon, Jun 12, 2023 at 7:16 PM grin  wrote:
>
> Hello,
>
>
> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy
>
> (stable)
>
>
> There is a single (test) radosgw serving plenty of test traffic. When
>
> under heavy req/s ("heavy" in a low sense, about 1k rq/s) it pretty
>
> reliably hangs: low traffic threads seem to work (like handling occasional
>
> PUTs) but GETs are completely nonresponsive, all attention seems to be
>
> spent on futexes.
>
>
> The effect is extremely similar to
>
>
>
> https://ceph-users.ceph.narkive.com/I4uFVzH9/radosgw-civetweb-hangs-once-around-850-established-connections
>
> (subject: Radosgw (civetweb) hangs once around)
>
> except this is quincy so it's beast instead of civetweb. The effect is the
>
> same as described there, except the cluster is way smaller (about 20-40
>
> OSDs).
>
>
> I observed that when I start radosgw -f with debug 20/20 it almost never
>
> hangs, so my guess is some ugly race condition. However I am a bit clueless
>
> how to actually debug it since debugging makes it go away. Debug 1
>
> (default) with -d seems to hang after a while but it's not that simple to
>
> induce, I'm still testing under 4/4.
>
>
> Also I do not see much to configure about beast.
>
>
> As to answer the question in the original (2016) thread:
>
> - Debian stable
>
> - no visible limits issue
>
> - no obvious memory leak observed
>
> - no other visible resource shortage
>
> - strace says everyone's waiting on futexes, about 600-800 threads, apart
>
> from the one serving occasional PUTs
>
> - tcp port doesn't respond.
>
>
> IRC didn't react. ;-)
>
>
> Thanks,
>
> Peter
>
> ___
>
> ceph-users mailing list -- ceph-users@ceph.io
>
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> --
> This message is confidential and is for the sole use of the intended
> recipient(s). It may also be privileged or otherwise protected by copyright
> or other legal rules. If you have received it by mistake please let us know
> by reply email and delete it from your system. It is prohibited to copy
> this message or disclose its content to anyone. Any confidentiality or
> privilege is not waived or lost by any mistaken delivery or unauthorized
> disclosure of the message. All messages sent to and from Agoda may be
> monitored to ensure compliance with company policies, to protect the
> company's interests and to remove potential malware. Electronic messages
> may be intercepted, amended, lost or deleted, or contain viruses.
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: radosgw hang under pressure

[Rok Jaklič]

We are experiencing something similar (slow GETs responses) when sending 1k
delete requests for example in ceph v16.2.13.

Rok

On Mon, Jun 12, 2023 at 7:16 PM grin  wrote:

> Hello,
>
> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy
> (stable)
>
> There is a single (test) radosgw serving plenty of test traffic. When
> under heavy req/s ("heavy" in a low sense, about 1k rq/s) it pretty
> reliably hangs: low traffic threads seem to work (like handling occasional
> PUTs) but GETs are completely nonresponsive, all attention seems to be
> spent on futexes.
>
> The effect is extremely similar to
>
> https://ceph-users.ceph.narkive.com/I4uFVzH9/radosgw-civetweb-hangs-once-around-850-established-connections
> (subject: Radosgw (civetweb) hangs once around)
> except this is quincy so it's beast instead of civetweb. The effect is the
> same as described there, except the cluster is way smaller (about 20-40
> OSDs).
>
> I observed that when I start radosgw -f with debug 20/20 it almost never
> hangs, so my guess is some ugly race condition. However I am a bit clueless
> how to actually debug it since debugging makes it go away. Debug 1
> (default) with -d seems to hang after a while but it's not that simple to
> induce, I'm still testing under 4/4.
>
> Also I do not see much to configure about beast.
>
> As to answer the question in the original (2016) thread:
> - Debian stable
> - no visible limits issue
> - no obvious memory leak observed
> - no other visible resource shortage
> - strace says everyone's waiting on futexes, about 600-800 threads, apart
> from the one serving occasional PUTs
> - tcp port doesn't respond.
>
> IRC didn't react. ;-)
>
> Thanks,
> Peter
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] RGW: exposing multi-tenant

[Rok Jaklič]

Hi,

are there any drawbacks of exposing multi-tenant deployment of RGWs
directly to users so they can use any S3 client to connect to service or
should we put something in front of RGWs?

How many users in multi-tenant deployment can CEPH handle?

Kind regards,
Rok
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Dedicated radosgw gateways

[Rok Jaklič]

I've searched for rgw_enable_lc_threads and rgw_enable_gc_threads a bit.

but there is little information about those settings. Is there any
documentation in the wild about those settings?

Are they enabled by default?



On Thu, May 18, 2023 at 9:15 PM Tarrago, Eli (RIS-BCT) <
eli.tarr...@lexisnexisrisk.com> wrote:

> Adding a bit more context to this thread.
>
> I added an additional radosgw to each cluser. Radosgw 1-3 are customer
> facing. Radosgw #4 is dedicated to syncing
>
> Radosgw 1-3 now have an additional lines:
> rgw_enable_lc_threads = False
> rgw_enable_gc_threads = False
>
> Radosgw4 has the additional line:
> rgw_sync_obj_etag_verify = True
>
> The logs on any of the radosgw’s appear to be identical, but here is an
> example log that is reflect on any of the servers. Notice the IP addresses
> are 1-4. Where I expect the traffic to be from only 4.
>
> Is this to be expected?
>
> My understanding of this thread is that this traffic would be regulated to
> radosgw 04.
>
>
> Example Ceph Conf for a single node, this is RadosGw 01
>
> [client.rgw.west01.rgw0]
> host = west01
> keyring = /var/lib/ceph/radosgw/west-rgw.west01.rgw0/keyring
> log file = /var/log/ceph/west-rgw-west01.rgw0.log
> rgw frontends = beast port=8080 num_threads=500
> rgw_dns_name = west01.example.com
> rgw_max_chunk_size = 67108864
> rgw_obj_stripe_size = 67108864
> rgw_put_obj_min_window_size = 67108864
> rgw_zone = rgw-west
> rgw_enable_lc_threads = False
> rgw_enable_gc_threads = False
>
> 
>
> Example Logs:
>
> 2023-05-18T19:06:48.295+ 7fb295f83700  1 beast: 0x7fb3e82b26f0:
> 10.10.10.1 - synchronization-user [18/May/2023:19:06:48.107 +] "GET
> /admin/log/?type=data&id=69&marker=1_&extra-info=true&rgwx-zonegroup=x
> HTTP/1.1" 200 44 - - - latency=0.188007131s
> 2023-05-18T19:06:48.371+ 7fb1dd612700  1 == starting new request
> req=0x7fb3e80ae6f0 =
> 2023-05-18T19:06:48.567+ 7fb1dd612700  1 == req done
> req=0x7fb3e80ae6f0 op status=0 http_status=200 latency=0.196007445s ==
> 2023-05-18T19:06:48.567+ 7fb1dd612700  1 beast: 0x7fb3e80ae6f0:
> 10.10.10.3 - synchronization-user [18/May/2023:19:06:48.371 +] "GET
> /admin/log/?type=data&id=107&marker=1_&extra-info=true&rgwx-zonegroup=
> HTTP/1.1" 200 44 - - - latency=0.196007445s
> 2023-05-18T19:06:49.023+ 7fb290f79700  1 == starting new request
> req=0x7fb3e81b06f0 =
> 2023-05-18T19:06:49.023+ 7fb28bf6f700  1 == req done
> req=0x7fb3e81b06f0 op status=0 http_status=200 latency=0.0s ==
> 2023-05-18T19:06:49.023+ 7fb28bf6f700  1 beast: 0x7fb3e81b06f0:
> 10.10.10.2 - synchronization-user [18/May/2023:19:06:49.023 +] "GET
> /admin/log?bucket-instance=ceph-bucketxxx%3A81&format=json&marker=0020447.3609723.6&type=bucket-index&rgwx-zonegroup=xxx
> HTTP/1.1" 200 2 - - - latency=0.0s
> 2023-05-18T19:06:49.147+ 7fb27af4d700  1 == starting new request
> req=0x7fb3e81b06f0 =
> 2023-05-18T19:06:49.151+ 7fb27af4d700  1 == req done
> req=0x7fb3e81b06f0 op status=0 http_status=200 latency=0.004000151s ==
> 2023-05-18T19:06:49.475+ 7fb280f59700  1 beast: 0x7fb3e82316f0:
> 10.10.10.4 - synchronization-user [18/May/2023:19:06:49.279 +] "GET
> /admin/log/?type=data&id=58&marker=1_.1&extra-info=true&rgwx-zonegroup=
> HTTP/1.1" 200 312 - - - latency=0.196007445s
> 2023-05-18T19:06:49.987+ 7fb27c750700  1 == starting new request
> req=0x7fb3e81b06f0 =
>
>
>
>
> radosgw-admin zonegroup get
> {
> "id": "x",
> "name": "eastWestceph",
> "api_name": "EastWestCeph",
> "is_master": "true",
> "endpoints": [
> "http://east01.noam.lnrm.net:8080";,
> "http://east02.noam.lnrm.net:8080";,
> "http://east03.noam.lnrm.net:8080";,
> "http://east04.noam.lnrm.net:8080";, <<  sync node
> "http://west01.noam.lnrm.net:8080";,
> "http://west02.noam.lnrm.net:8080";,
> "http://west03.noam.lnrm.net:8080";,
> "http://west04.noam.lnrm.net:8080"; <<  sync node
> ],
> ...
> ],
> "hostnames_s3website": [],
> "master_zone": "x",
> "zones": [
> {
> "id": "x",
> "name": "rgw-west",
> "endpoints": [
> "http://west01.noam.lnrm.net:8080";,
> "http://west02.noam.lnrm.net:8080";,
> "http://west03.noam.lnrm.net:8080";,
> "http://west04.noam.lnrm.net:8080"; << -- sync node
> ],
> "log_meta": "false",
> "log_data": "true",
> "bucket_index_max_shards": 0,
> "read_only": "false",
> "tier_type": "",
> "sync_from_all": "true",
> "sync_from": [],
> "redirect_zone": ""
> },
> {
> "id": "x",
> "name": "rgw-east",
> "endpoints": [
> "http://east01.noam.lnrm.net:8080";,
>

[ceph-users] Re: Deleting millions of objects

[Rok Jaklič]

Thx for the input.

I tried several config sets e.g.:
ceph config set client.radosgw.mon2 rgw_delete_multi_obj_max_num 1
ceph config set client.radosgw.mon1 rgw_delete_multi_obj_max_num 1
ceph config set client.rgw rgw_delete_multi_obj_max_num 1

where client.radosgw.mon2 is the same as in ceph.conf but without success.

It also seems from
https://github.com/ceph/ceph/blob/8c4f52415bddba65e654f3a4f7ba37d98446d202/src/rgw/rgw_op.cc#L7131
that it should check config setting, but for some reason it is not working.

---

For now I ended up with spawning up to 100 background processes (more than
that it fills up our FE queue and we get response timeouts) with:
mc rm --recursive --force ceph/archive/veeam &

Regards,
Rok

On Thu, May 18, 2023 at 3:47 AM Szabo, Istvan (Agoda) <
istvan.sz...@agoda.com> wrote:

> If it works I’d be amazed. We have this slow and limited delete issue
> also. What we’ve done to run on the same bucket multiple delete from
> multiple servers via s3cmd.
>
> Istvan Szabo
> Staff Infrastructure Engineer
> ---
> Agoda Services Co., Ltd.
> e: istvan.sz...@agoda.com
> ---
>
> On 2023. May 17., at 20:14, Joachim Kraftmayer - ceph ambassador <
> joachim.kraftma...@clyso.com> wrote:
>
> Email received from the internet. If in doubt, don't click any link nor
> open any attachment !
> 
>
> Hi Rok,
>
> try this:
>
>
> rgw_delete_multi_obj_max_num - Max number of objects in a single
> multi-object delete request
>  (int, advanced)
>  Default: 1000
>  Can update at runtime: true
>  Services: [rgw]
>
>
> config set   
>
>
> WHO: client. or client.rgw
>
> KEY: rgw_delete_multi_obj_max_num
>
> VALUE: 1
>
> Regards, Joachim
>
> ___
> ceph ambassador DACH
> ceph consultant since 2012
>
> Clyso GmbH - Premier Ceph Foundation Member
>
> https://www.clyso.com/
>
> Am 17.05.23 um 14:24 schrieb Rok Jaklič:
>
> thx.
>
>
> I tried with:
>
> ceph config set mon rgw_delete_multi_obj_max_num 1
>
> ceph config set client rgw_delete_multi_obj_max_num 1
>
> ceph config set global rgw_delete_multi_obj_max_num 1
>
>
> but still only 1000 objects get deleted.
>
>
> Is the target something different?
>
>
> On Wed, May 17, 2023 at 11:58 AM Robert Hish 
>
> wrote:
>
>
> I think this is capped at 1000 by the config setting. Ive used the aws
>
> and s3cmd clients to delete more than 1000 objects at a time and it
>
> works even with the config setting capped at 1000. But it is a bit slow.
>
>
> #> ceph config help rgw_delete_multi_obj_max_num
>
>
> rgw_delete_multi_obj_max_num - Max number of objects in a single multi-
>
> object delete request
>
>   (int, advanced)
>
>   Default: 1000
>
>   Can update at runtime: true
>
>   Services: [rgw]
>
>
> On Wed, 2023-05-17 at 10:51 +0200, Rok Jaklič wrote:
>
> Hi,
>
>
> I would like to delete millions of objects in RGW instance with:
>
> mc rm --recursive --force ceph/archive/veeam
>
>
> but it seems it allows only 1000 (or 1002 exactly) removals per
>
> command.
>
>
> How can I delete/remove all objects with some prefix?
>
>
> Kind regards,
>
> Rok
>
> ___
>
> ceph-users mailing list -- ceph-users@ceph.io
>
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> ___
>
> ceph-users mailing list -- ceph-users@ceph.io
>
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> ___
>
> ceph-users mailing list -- ceph-users@ceph.io
>
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> --
> This message is confidential and is for the sole use of the intended
> recipient(s). It may also be privileged or otherwise protected by copyright
> or other legal rules. If you have received it by mistake please let us know
> by reply email and delete it from your system. It is prohibited to copy
> this message or disclose its content to anyone. Any confidentiality or
> privilege is not waived or lost by any mistaken delivery or unauthorized
> disclosure of the message. All messages sent to and from Agoda may be
> monitored to ensure compliance with company policies, to protect the
> company's interests and to remove potential malware. Electronic messages
> may be intercepted, amended, lost or deleted, or contain viruses.
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Deleting millions of objects

[Rok Jaklič]

thx.

I tried with:
ceph config set mon rgw_delete_multi_obj_max_num 1
ceph config set client rgw_delete_multi_obj_max_num 1
ceph config set global rgw_delete_multi_obj_max_num 1

but still only 1000 objects get deleted.

Is the target something different?

On Wed, May 17, 2023 at 11:58 AM Robert Hish 
wrote:

>
> I think this is capped at 1000 by the config setting. Ive used the aws
> and s3cmd clients to delete more than 1000 objects at a time and it
> works even with the config setting capped at 1000. But it is a bit slow.
>
> #> ceph config help rgw_delete_multi_obj_max_num
>
> rgw_delete_multi_obj_max_num - Max number of objects in a single multi-
> object delete request
>   (int, advanced)
>   Default: 1000
>   Can update at runtime: true
>   Services: [rgw]
>
> On Wed, 2023-05-17 at 10:51 +0200, Rok Jaklič wrote:
> > Hi,
> >
> > I would like to delete millions of objects in RGW instance with:
> > mc rm --recursive --force ceph/archive/veeam
> >
> > but it seems it allows only 1000 (or 1002 exactly) removals per
> > command.
> >
> > How can I delete/remove all objects with some prefix?
> >
> > Kind regards,
> > Rok
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Deleting millions of objects

[Rok Jaklič]

Hi,

I would like to delete millions of objects in RGW instance with:
mc rm --recursive --force ceph/archive/veeam

but it seems it allows only 1000 (or 1002 exactly) removals per command.

How can I delete/remove all objects with some prefix?

Kind regards,
Rok
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Moving From BlueJeans to Jitsi for Ceph meetings

[Rok Jaklič]

We deployed jitsi for the public sector during covid and it is still free
to use.

https://vid.arnes.si/

---

However, the landing page is in Slovene language and for future
reservations you need an AAI (SSO) account (which you get if you are a part
of a public organization (school, faculty, ...).

We've noticed that for conferences of over >20 people it used to not work
well for some people (those with slow internet connection) but do not know
if this problem still exists.

Rok

On Wed, Mar 22, 2023 at 6:24 AM Alvaro Soto  wrote:

> +1 jitsi
>
> ---
> Alvaro Soto.
>
> Note: My work hours may not be your work hours. Please do not feel the need
> to respond during a time that is not convenient for you.
> --
> Great people talk about ideas,
> ordinary people talk about things,
> small people talk... about other people.
>
> On Tue, Mar 21, 2023, 2:02 PM Federico Lucifredi 
> wrote:
>
> > Jitsi is really good, and getting better — we have been using it with my
> > local User’s Group for the last couple of years.
> >
> > Only observation is to discover the maximum allowable number of guests in
> > advance if this is not already known - we had a fairly generous allowance
> > in BlueJeans accounts for Red Hat, Jitsi community accounts may not be as
> > large.
> >
> > Best-F
> >
> > > On Mar 21, 2023, at 12:26, Mike Perez  wrote:
> > >
> > > I'm not familiar with BBB myself. Are there any objections to Jitsi? I
> > > want to update the calendar invites this week.
> > >
> > >> On Thu, Mar 16, 2023 at 6:16 PM huxia...@horebdata.cn
> > >>  wrote:
> > >>
> > >> Besides Jitsi, another option would be BigBlueButton(BBB). Does anyone
> > know how BBB compares with Jitsi?
> > >>
> > >>
> > >>
> > >>
> > >> huxia...@horebdata.cn
> > >>
> > >> From: Mike Perez
> > >> Date: 2023-03-16 21:54
> > >> To: ceph-users
> > >> Subject: [ceph-users] Moving From BlueJeans to Jitsi for Ceph meetings
> > >> Hi everyone,
> > >>
> > >> We have been using BlueJeans to meet and record some of our meetings
> > >> that later get posted to our YouTube channel. Unfortunately, we have
> > >> to figure out a new meeting platform due to Red Hat discontinuing
> > >> BlueJeans by the end of this month.
> > >>
> > >> Google Meets is an option, but some users in other countries have
> > >> trouble using Google's services.
> > >>
> > >> For some meetings, we have tried out Jitsi, and it works well, meets
> > >> our requirements, and is free.
> > >>
> > >> Does anyone else have suggestions for another free meeting platform
> > >> that provides recording capabilities?
> > >>
> > >> --
> > >> Mike Perez
> > >> ___
> > >> ceph-users mailing list -- ceph-users@ceph.io
> > >> To unsubscribe send an email to ceph-users-le...@ceph.io
> > >>
> > >> ___
> > >> ceph-users mailing list -- ceph-users@ceph.io
> > >> To unsubscribe send an email to ceph-users-le...@ceph.io
> > >>
> > >
> > >
> > > --
> > > Mike Perez
> > > ___
> > > ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to ceph-users-le...@ceph.io
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: rbd on EC pool with fast and extremely slow writes/reads

[Rok Jaklič]

1, 2 times a year we are having similar problem in *not* ceph disk cluster,
where working -> but slow disk writes give us slow reads. We somehow
"understand it", since probably slow writes fill up queues and buffers.


On Thu, Mar 9, 2023 at 11:37 AM Andrej Filipcic 
wrote:

>
> Thanks for the hint, did run some short test, all fine. I am not sure
> it's a drive issue.
>
> Some more digging, the file with bad performance has this segments:
>
> [root@afsvos01 vicepa]# hdparm --fibmap $PWD/0
>
> /vicepa/0:
> filesystem blocksize 4096, begins at LBA 2048; assuming 512 byte sectors.
> byte_offset  begin_LBAend_LBAsectors
>0 74323228150392071808
>   1060765696373306458382792105216
>   2138636288   70841232   87586575   16745344
> 10712252416   87586576   87635727  49152
>
> Reading by segments:
>
> # dd if=0 of=/tmp/0 bs=4M status=progress count=252
> 1052770304 bytes (1.1 GB, 1004 MiB) copied, 45 s, 23.3 MB/s
> 252+0 records in
> 252+0 records out
>
> # dd if=0 of=/tmp/0 bs=4M status=progress skip=252 count=256
> 935329792 bytes (935 MB, 892 MiB) copied, 4 s, 234 MB/s
> 256+0 records in
> 256+0 records out
>
> # dd if=0 of=/tmp/0 bs=4M status=progress skip=510
> 7885291520 bytes (7.9 GB, 7.3 GiB) copied, 12 s, 657 MB/s
> 2050+0 records in
> 2050+0 records out
>
> So, 1st 1G is very slow, second segment is faster, then the rest quite
> fast, and it's reproducible (dropped caches before each dd)
>
> Now, the rbd is 3TB with 256 pgs (EC 8+3), I checked with rados that
> objects are randomly distributed on pgs, eg
>
> # rados --pgid 23.82 ls|grep rbd_data.20.2723bd3292f6f8
> rbd_data.20.2723bd3292f6f8.0008
> rbd_data.20.2723bd3292f6f8.000d
> rbd_data.20.2723bd3292f6f8.01cb
> rbd_data.20.2723bd3292f6f8.000601b2
> rbd_data.20.2723bd3292f6f8.0009001b
> rbd_data.20.2723bd3292f6f8.005b
> rbd_data.20.2723bd3292f6f8.000900e8
>
> where object ...05b for example corresponds to the 1st block of the file
> I am testing. Well, if my understanding of rbd  is correct: I assume
> that LBA regions are mapped to consecutive rbd objects.
>
> So, now I am completely confused since the slow chunk of the file is
> still mapped to ~256 objects on different pgs
>
> Maybe I misunderstood the whole thing.
>
> Any other hints? we will still do hdd tests on all the drives
>
> Cheers,
> Andrej
>
> On 3/6/23 20:25, Paul Mezzanini wrote:
> > When I have seen behavior like this it was a dying drive.  It only
> became obviously when I did a smart long test and I got failed reads.
> Still reported smart OK though so that was a lie.
> >
> >
> >
> > --
> >
> > Paul Mezzanini
> > Platform Engineer III
> > Research Computing
> >
> > Rochester Institute of Technology
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > 
> > From: Andrej Filipcic
> > Sent: Monday, March 6, 2023 8:51 AM
> > To: ceph-users
> > Subject: [ceph-users] rbd on EC pool with fast and extremely slow
> writes/reads
> >
> >
> > Hi,
> >
> > I have a problem on one of ceph clusters I do not understand.
> > ceph 17.2.5 on 17 servers, 400 HDD OSDs, 10 and 25Gb/s NICs
> >
> > 3TB rbd image is on erasure coded 8+3 pool with 128pgs , xfs filesystem,
> > 4MB objects in rbd image, mostly empy.
> >
> > I have created a bunch of 10G files, most of them were written with
> > 1.5GB/s, few of them were really slow, ~10MB/s, a factor of 100.
> >
> > When reading these files back, the fast-written ones are read fast,
> > ~2-2.5GB/s, the slowly-written are also extremely slow in reading, iotop
> > shows between 1 and 30 MB/s reading speed.
> >
> > This does not happen at all on replicated images. There are some OSDs
> > with higher apply/commit latency, eg 200ms, but there are no slow ops.
> >
> > The tests were done actually on proxmox vm with librbd, but the same
> > happens with krbd, and on bare metal with mounted krbd as well.
> >
> > I have tried to check all OSDs for laggy drives, but they all look about
> > the same.
> >
> > I have also copied entire image with "rados get...", object by object,
> > the strange thing here is that most of objects were copied within
> > 0.1-0.2s, but quite some took more than 1s.
> > The cluster is quite busy with base traffic of ~1-2GB/s, so the speeds
> > can vary due to that. But I would not expect a factor of 100 slowdown
> > for some writes/reads with rbds.
> >
> > Any clues on what might be wrong or what else to check? I have another
> > similar ceph cluster where everything looks fine.
> >
> > Best,
> > Andrej
> >
> > --
> > _
> >  prof. dr. Andrej Filipcic,   E-mail:andrej.filip...@ijs.si
> >  Department of Experimental High Energy Physics - F9
> >  Jozef Stefan Institute, Jamova 39, P.o.Box 3000
> >  SI-1001 Ljubljana, Slovenia
> >  Tel.: +386-1-477-3674Fax: +386-1-477-3166
> > --

[ceph-users] rgw with unix socket

[Rok Jaklič]

Hi,

I try to configure ceph with rgw and unix socket (based on
https://docs.ceph.com/en/pacific/man/8/radosgw/?highlight=radosgw). I have
in ceph.conf something like this:
[client.radosgw.ctplmon3]
host = ctplmon3
rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
log file = /var/log/ceph/client.radosgw.ctplmon3.log
rgw print continue = false

When I start radosgw with:
radosgw -c /etc/ceph/ceph.conf --setuser ceph --setgroup ceph -n
client.radosgw.ctplmon3

I get in logs:
2022-10-17T17:11:22.925+0200 7f75c72545c0  0 ceph version 16.2.10
(45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable), process
radosgw, pid 4077748
2022-10-17T17:11:22.925+0200 7f75c72545c0  0 framework: beast
2022-10-17T17:11:22.925+0200 7f75c72545c0  0 framework conf key: port, val:
7480
2022-10-17T17:11:22.925+0200 7f75c72545c0  1 radosgw_Main not setting numa
affinity
2022-10-17T17:11:24.863+0200 7f75c72545c0  0 framework: beast
2022-10-17T17:11:24.863+0200 7f75c72545c0  0 framework conf key:
ssl_certificate, val: config://rgw/cert/$realm/$zone.crt
2022-10-17T17:11:24.863+0200 7f75c72545c0  0 framework conf key:
ssl_private_key, val: config://rgw/cert/$realm/$zone.key
2022-10-17T17:11:24.863+0200 7f75c72545c0  0 starting handler: beast
2022-10-17T17:11:24.867+0200 7f75c72545c0  0 set uid:gid to 167:167
(ceph:ceph)
2022-10-17T17:11:24.904+0200 7f75c72545c0  1 mgrc service_daemon_register
rgw.11621227 metadata {arch=x86_64,ceph_release=pacific,ceph_version=ceph
version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific
(stable),ceph_version_short=16.2.10,cpu=Intel(R) Xeon(R) Silver 4114 CPU @
2.20GHz,distro=almalinux,distro_description=AlmaLinux 8.6 (Sky
Tiger),distro_version=8.6,frontend_config#0=beast
port=7480,frontend_type#0=beast,hostname=ctplmon3.arnes.si,id=radosgw.ctplmon3,kernel_description=#1
SMP Tue Aug 2 13:42:59 EDT
2022,kernel_version=4.18.0-372.19.1.el8_6.x86_64,mem_swap_kb=8388604,mem_total_kb=65325948,num_handles=1,os=Linux,pid=4077748,realm_id=,realm_name=,zone_id=c2c70444-7a41-4acd-a0d0-9f87d324ec72,zone_name=default,zonegroup_id=b1e0d55c-f7cb-4e73-b1cb-6cffa1fd6578,zonegroup_name=default}
2022-10-17T17:20:37.712+0200 7f75ae9a2700 -1 received  signal: Interrupt,
si_code : 128, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid :
0, si_uid : 0, si_addr0, si_status0

... where it seems it started rgw on port and ip.

Looking at:
https://github.com/ceph/ceph/blob/quincy/src/rgw/rgw_asio_frontend.cc

I do not see any reference to handling rgw on unix sockets. Is this even
implemented?

Kind regards,
Rok
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RGW problems after upgrade to 16.2.10

[Rok Jaklič]

Solution was found by colleague and it was:
ms_mon_client_mode = crc

... because of
https://github.com/ceph/ceph/pull/42587/commits/7e22d2a31d277ab3eecff47b0864b206a32e2332

Rok

On Thu, Sep 8, 2022 at 6:04 PM Rok Jaklič  wrote:

> What credentials should RGWs have?
>
> I have intentionally
>
> auth cluster required = none
> auth service required = none
> auth client required = none
>
> and in 16.2.7 it worked.
>
> Kind regards,
> Rok
>
> On Thu, Sep 8, 2022, 14:29 J. Eric Ivancich  wrote:
>
>> What jumps out to me is:
>>
>> a. The -13 error code represents permission denied
>> b. You’ve commented out the keyring configuration in ceph.conf
>>
>> So do your RGWs have appropriate credentials?
>>
>> Eric
>> (he/him)
>>
>> > On Sep 7, 2022, at 3:04 AM, Rok Jaklič  wrote:
>> >
>> > Hi,
>> >
>> > after upgrading to ceph version 16.2.10 from 16.2.7 rados gw is not
>> > working. We start rados gw with:
>> > radosgw -c /etc/ceph/ceph.conf --setuser ceph --setgroup ceph -n
>> > client.radosgw.ctplmon3
>> >
>> > ceph.conf looks like:
>> > [root@ctplmon3 ~]# cat /etc/ceph/ceph.conf
>> > [global]
>> > fsid = 0a6e5422-ac75-4093-af20-528ee00cc847
>> > mon initial members = ctplmon1,ctplmon3
>> > mon host = X_IP, Y_IP
>> > auth cluster required = none
>> > auth service required = none
>> > auth client required = none
>> > osd journal size = 1024
>> > osd pool default size = 3
>> > osd pool default min size = 2
>> > osd pool default pg num = 128
>> > osd pool default pgp num = 128
>> > osd crush chooseleaf type = 1
>> >
>> > [client.radosgw.ctplmon3]
>> > # keyring = /etc/ceph/keyring.radosgw.gateway
>> > host = ctplmon3
>> > log file = /var/log/ceph/client.radosgw.ctplmon3.log
>> > rgw_dns_name = Y_HOSTNAME
>> > rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
>> > rgw print continue = false
>> >
>> > And in logs we get:
>> > 2022-09-07T08:55:28.824+0200 7f03fe033700  0 --2- Y_IP:0/662532535 >>
>> > v2:Y_IP:3300/0 conn(0x5648b93bb000 0x5648b940a800 unknown :-1
>> > s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request
>> > get_initial_auth_request returned -13
>> > 2022-09-07T08:55:28.826+0200 7f03fe033700  0 --2- Y_IP:0/662532535 >>
>> > [v2:Y_IP:3300/0,v1:Y_IP:6789/0] conn(0x5648ba0b4000 0x5648b940ad00
>> unknown
>> > :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request
>> > get_initial_auth_request returned -13
>> > 2022-09-07T08:55:28.826+0200 7f03fd832700  0 --2- Y_IP:0/662532535 >>
>> > [v2:X_IP:3300/0,v1:X_IP:6789/0] conn(0x5648ba0b4400 0x5648b940b200
>> unknown
>> > :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request
>> > get_initial_auth_request returned -13
>> > 2022-09-07T08:55:28.826+0200 7f03fe834700  0 --2- Y_IP:0/662532535 >>
>> > [v2:Z_IP:3300/0,v1:Z_IP:6789/0] conn(0x5648ba0b4800 0x5648b940b700
>> unknown
>> > :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request
>> > get_initial_auth_request returned -13
>> >
>> > We have a firewall open and tcpdump shows traffic when rados tries to
>> > connect.
>> >
>> > Any ideas why rados cannot start?
>> >
>> > Kind regards,
>> > Rok
>> > ___
>> > ceph-users mailing list -- ceph-users@ceph.io
>> > To unsubscribe send an email to ceph-users-le...@ceph.io
>> >
>>
>>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Requested range is not satisfiable

[Rok Jaklič]

Hi,

we try to copy a big file (over 400GB) using a minio client to the ceph
cluster. Copy or better transfer takes a lot of time (2 days for example)
because of "slow connection".

Usually somewhere near the end (but looks random) we get an error like:
 Failed to copy `/360GB.bigfile.img`. The requested range is not
satisfiable

We are using RGW.

Any ideas why?

Kind regards,
Rok
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Manual deployment, documentation error?

[Rok Jaklič]

Every now and then someone comes up with a subject like this.

There is quite a long thread about pros and cons using docker and all tools
around ceph on
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/TTTYKRVWJOR7LOQ3UCQAZQR32R7YADVY/#AT7YQV6RE5SMKDZHXL3ZI2G5BWFUUUXE

Long story short... additional layers of complexity underneath and
simplicity on the top because of some hype around docker or something does
not solve problems users are facing right now. So the argument for ceph not
using docker in installation is actually quite good.

On Thu, Sep 15, 2022 at 10:18 AM Dominique Ramaekers <
dominique.ramaek...@cometal.be> wrote:

> Hi Ranjan,
>
> I don't want to intervene but I can testify that docker doesn't make the
> installation for a 3-node cluster overkill.
>
> I to have a 3-node cluster (to be expanded soon to 4 nodes).
>
> Cons of using cephadm (and thus docker):
> - You need to learn the basics of docker
>
> Pros:
> + cephadm works very easy. The time you spend on learing docker will be
> easely compensated by the small time you need to learn cephadm
> + Upgrading a manual installation is very tricky! Cephadm manages upgrades
> of ceph automatically. You only need to give the command (done it already
> two times).
> + If you need to upgrade your OS, will the manual installation still
> function? With cephadm the ceph processes inside the docker containers
> experience minimal impact with the upgrade of the os (dind't do an OS
> upgrade yet, but had this issue with other applications).
>
> Greetings,
>
> Dominique.
>
> > -Oorspronkelijk bericht-
> > Van: Ranjan Ghosh 
> > Verzonden: woensdag 14 september 2022 15:58
> > Aan: Eugen Block 
> > CC: ceph-users@ceph.io
> > Onderwerp: [ceph-users] Re: Manual deployment, documentation error?
> >
> > Hi Eugen,
> >
> > thanks for your answer. I don't want to use the cephadm tool because it
> > needs docker. I don't like it because it's total overkill for our small
> 3-node
> > cluster.  I'd like to avoid the added complexity, added packages,
> everything.
> > Just another thing I have to learn in detaisl about in case things go
> wrong.
> >
> > The monitor service is running but the logs don't say anything :-( Okay,
> but at
> > least I know now that it should work in principle without the mgr.
> >
> > Thank you
> > Ranjan
> >
> > Eugen Block schrieb am 14.09.2022 um 15:26:
> > > Hi,
> > >
> > >> Im currently trying the manual deployment because ceph-deploy
> > >> unfortunately doesn't seem to exist anymore and under step 19 it says
> > >> you should run "sudo ceph -s". That doesn't seem to work. I guess
> > >> this is because the manager service isn't yet running, right?
> > >
> > > ceph-deploy was deprecated quite some time ago, if you want to use a
> > > deployment tool try cephadm [1]. The command 'ceph -s' is not
> > > depending on the mgr but the mon service. So if that doesn't work you
> > > need to check the mon logs and see if the mon service is up and
> running.
> > >
> > >> Interestingly the screenshot under step 19 says "mgr: mon-
> > >> node1(active)". If you follow the documentation step by step, there's
> > >> no mention of the manager node up until that point.
> > >
> > > Right after your mentioned screenshot there's a section for the mgr
> > > service [2].
> > >
> > > Regards,
> > > Eugen
> > >
> > > [1] https://docs.ceph.com/en/quincy/cephadm/install/
> > > [2]
> > > https://docs.ceph.com/en/quincy/install/manual-deployment/#manager-
> > dae
> > > mon-configuration
> > >
> > >
> > > Zitat von Ranjan Ghosh :
> > >
> > >> Hi all,
> > >>
> > >> I think there's an error in the documentation:
> > >>
> > >> https://docs.ceph.com/en/quincy/install/manual-deployment/
> > >>
> > >> Im currently trying the manual deployment because ceph-deploy
> > >> unfortunately doesn't seem to exist anymore and under step 19 it says
> > >> you should run "sudo ceph -s". That doesn't seem to work. I guess
> > >> this is because the manager service isn't yet running, right?
> > >>
> > >> Interestingly the screenshot under step 19 says "mgr: mon-
> > >> node1(active)". If you follow the documentation step by step, there's
> > >> no mention of the manager node up until that point.
> > >>
> > >> Thank you / BR
> > >> Ranjan
> > >>
> > >> ___
> > >> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
> > >> email to ceph-users-le...@ceph.io
> > >
> > >
> > >
> > > ___
> > > ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
> > > email to ceph-users-le...@ceph.io
> >
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
> email
> > to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
__

[ceph-users] Re: RGW problems after upgrade to 16.2.10

[Rok Jaklič]

What credentials should RGWs have?

I have intentionally

auth cluster required = none
auth service required = none
auth client required = none

and in 16.2.7 it worked.

Kind regards,
Rok

On Thu, Sep 8, 2022, 14:29 J. Eric Ivancich  wrote:

> What jumps out to me is:
>
> a. The -13 error code represents permission denied
> b. You’ve commented out the keyring configuration in ceph.conf
>
> So do your RGWs have appropriate credentials?
>
> Eric
> (he/him)
>
> > On Sep 7, 2022, at 3:04 AM, Rok Jaklič  wrote:
> >
> > Hi,
> >
> > after upgrading to ceph version 16.2.10 from 16.2.7 rados gw is not
> > working. We start rados gw with:
> > radosgw -c /etc/ceph/ceph.conf --setuser ceph --setgroup ceph -n
> > client.radosgw.ctplmon3
> >
> > ceph.conf looks like:
> > [root@ctplmon3 ~]# cat /etc/ceph/ceph.conf
> > [global]
> > fsid = 0a6e5422-ac75-4093-af20-528ee00cc847
> > mon initial members = ctplmon1,ctplmon3
> > mon host = X_IP, Y_IP
> > auth cluster required = none
> > auth service required = none
> > auth client required = none
> > osd journal size = 1024
> > osd pool default size = 3
> > osd pool default min size = 2
> > osd pool default pg num = 128
> > osd pool default pgp num = 128
> > osd crush chooseleaf type = 1
> >
> > [client.radosgw.ctplmon3]
> > # keyring = /etc/ceph/keyring.radosgw.gateway
> > host = ctplmon3
> > log file = /var/log/ceph/client.radosgw.ctplmon3.log
> > rgw_dns_name = Y_HOSTNAME
> > rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
> > rgw print continue = false
> >
> > And in logs we get:
> > 2022-09-07T08:55:28.824+0200 7f03fe033700  0 --2- Y_IP:0/662532535 >>
> > v2:Y_IP:3300/0 conn(0x5648b93bb000 0x5648b940a800 unknown :-1
> > s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request
> > get_initial_auth_request returned -13
> > 2022-09-07T08:55:28.826+0200 7f03fe033700  0 --2- Y_IP:0/662532535 >>
> > [v2:Y_IP:3300/0,v1:Y_IP:6789/0] conn(0x5648ba0b4000 0x5648b940ad00
> unknown
> > :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request
> > get_initial_auth_request returned -13
> > 2022-09-07T08:55:28.826+0200 7f03fd832700  0 --2- Y_IP:0/662532535 >>
> > [v2:X_IP:3300/0,v1:X_IP:6789/0] conn(0x5648ba0b4400 0x5648b940b200
> unknown
> > :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request
> > get_initial_auth_request returned -13
> > 2022-09-07T08:55:28.826+0200 7f03fe834700  0 --2- Y_IP:0/662532535 >>
> > [v2:Z_IP:3300/0,v1:Z_IP:6789/0] conn(0x5648ba0b4800 0x5648b940b700
> unknown
> > :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request
> > get_initial_auth_request returned -13
> >
> > We have a firewall open and tcpdump shows traffic when rados tries to
> > connect.
> >
> > Any ideas why rados cannot start?
> >
> > Kind regards,
> > Rok
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] RGW problems after upgrade to 16.2.10

[Rok Jaklič]

Hi,

after upgrading to ceph version 16.2.10 from 16.2.7 rados gw is not
working. We start rados gw with:
radosgw -c /etc/ceph/ceph.conf --setuser ceph --setgroup ceph -n
client.radosgw.ctplmon3

ceph.conf looks like:
[root@ctplmon3 ~]# cat /etc/ceph/ceph.conf
[global]
fsid = 0a6e5422-ac75-4093-af20-528ee00cc847
mon initial members = ctplmon1,ctplmon3
mon host = X_IP, Y_IP
auth cluster required = none
auth service required = none
auth client required = none
osd journal size = 1024
osd pool default size = 3
osd pool default min size = 2
osd pool default pg num = 128
osd pool default pgp num = 128
osd crush chooseleaf type = 1

[client.radosgw.ctplmon3]
# keyring = /etc/ceph/keyring.radosgw.gateway
host = ctplmon3
log file = /var/log/ceph/client.radosgw.ctplmon3.log
rgw_dns_name = Y_HOSTNAME
rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
rgw print continue = false

And in logs we get:
2022-09-07T08:55:28.824+0200 7f03fe033700  0 --2- Y_IP:0/662532535 >>
v2:Y_IP:3300/0 conn(0x5648b93bb000 0x5648b940a800 unknown :-1
s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request
get_initial_auth_request returned -13
2022-09-07T08:55:28.826+0200 7f03fe033700  0 --2- Y_IP:0/662532535 >>
[v2:Y_IP:3300/0,v1:Y_IP:6789/0] conn(0x5648ba0b4000 0x5648b940ad00 unknown
:-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request
get_initial_auth_request returned -13
2022-09-07T08:55:28.826+0200 7f03fd832700  0 --2- Y_IP:0/662532535 >>
[v2:X_IP:3300/0,v1:X_IP:6789/0] conn(0x5648ba0b4400 0x5648b940b200 unknown
:-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request
get_initial_auth_request returned -13
2022-09-07T08:55:28.826+0200 7f03fe834700  0 --2- Y_IP:0/662532535 >>
[v2:Z_IP:3300/0,v1:Z_IP:6789/0] conn(0x5648ba0b4800 0x5648b940b700 unknown
:-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request
get_initial_auth_request returned -13

We have a firewall open and tcpdump shows traffic when rados tries to
connect.

Any ideas why rados cannot start?

Kind regards,
Rok
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Wrong size actual?

[Rok Jaklič]

Thanks for the info.

Is there any bug report open?

On Mon, Sep 5, 2022 at 4:44 PM Ulrich Klein 
wrote:

> Looks like the old problem of lost multipart upload fragments. Has been
> haunting me in all versions for more than a year. Haven‘t found any way of
> getting rid of them.
> Even deleting the bucket seems to leave the objects in the rados pool
> forever.
>
> Ciao, Uli
>
> > Am 05.09.2022 um 15:19 schrieb Rok Jaklič :
> >
> > Hi,
> >
> > when I do:
> > radosgw-admin user stats --uid=X --tenant=Y --sync-stats
> >
> > I get:
> > {
> >"stats": {
> >"size": 2620347285776,
> >"size_actual": 2620348436480,
> >"size_utilized": 0,
> >"size_kb": 2558932897,
> >"size_kb_actual": 2558934020,
> >"size_kb_utilized": 0,
> >"num_objects": 593
> >},
> >"last_stats_sync": "2022-09-05T13:11:05.009648Z",
> >"last_stats_update": "2022-09-05T13:11:04.97Z"
> > }
> >
> > even though I have only two empty buckets inside account. Any ideas why?
> >
> > Account was previously full of objects, but objects were delete few days
> > ago.
> >
> > Kind regards,
> > Rok
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Wrong size actual?

[Rok Jaklič]

Hi,

when I do:
radosgw-admin user stats --uid=X --tenant=Y --sync-stats

I get:
{
"stats": {
"size": 2620347285776,
"size_actual": 2620348436480,
"size_utilized": 0,
"size_kb": 2558932897,
"size_kb_actual": 2558934020,
"size_kb_utilized": 0,
"num_objects": 593
},
"last_stats_sync": "2022-09-05T13:11:05.009648Z",
"last_stats_update": "2022-09-05T13:11:04.97Z"
}

even though I have only two empty buckets inside account. Any ideas why?

Account was previously full of objects, but objects were delete few days
ago.

Kind regards,
Rok
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Tenant and user id

[Rok Jaklič]

Hi,

is it possible to get tenant and user id with some python boto3 request?

Kind regards,
Rok
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] reducing mon_initial_members

[Rok Jaklič]

Can I reduce mon_initial_members to one host after already being set to two
hosts?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: we're living in 2005.

[Rok Jaklič]

Actually, some of us tried to contribute to documentation but were stopped
with failed build checks for some reason.

While most of it is ok, at some places documentation is vague or missing
(maybe also the reason why this thread is so long also).

One example:
https://github.com/ceph/ceph/pull/40965/commits/9bd1fc55f73b4d243de018ecd0939543f950

On Tue, Jul 27, 2021 at 10:21 AM Wido den Hollander  wrote:

>
>
> Op 27-07-2021 om 05:11 schreef Fyodor Ustinov:
> > Hi!
> >
> >> docs.ceph.io ?  If there’s something that you’d like to see added
> there, you’re
> >> welcome to submit a tracker ticket, or write to me privately.  It is not
> >> uncommon for documentation enhancements to be made based on mailing list
> >> feedback.
> >
> > Documentation...
> >
> > Try to install a completely new ceph cluster from scratch on fresh
> installed LTS Ubuntu by this doc
> https://docs.ceph.com/en/latest/cephadm/install/ . Many interesting
> discoveries await you.
> > Nothing special - only step by step installation. As described in
> documentation. No more and no less.
> >
>
> But who's responsibility is it to write documentation? Ceph is Open
> Source and anybody can help to make it better.
>
> Developing Ceph is not only writing code, but that could also be writing
> documentation.
>
> Open Source != Free, that means that you need to invest time to get it
> working.
>
> And if you spot flaws in the documentation anybody is more then welcome
> to open a pull request to improve the documentation.
>
> Who else do we expect to write the documentation? In the end everybody
> needs to get paid and in the end the company who pays that person needs
> to get paid. That's how it works.
>
> Suggestions and feedback are always welcome, but don't expect a paved
> road for free.
>
> Wido
>
> > WBR,
> >  Fyodor.
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Limiting subuser to his bucket

[Rok Jaklič]

Hi,

is it possible to limit access of the subuser that he sees (read, write)
only "his" bucket? And also be able to create a bucket inside that bucket?

Kind regards,
Rok
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Rok Jaklič]

This thread would not be so long if docker/containers solved the problems,
but it did not. It solved some, but introduced new ones. So we cannot
really say its better now.

Again, I think focus should more on a working ceph with clean documentation
while leaving software management, packages to admins. And staticilly
linked binaries would certinly solve dependecy hell and "how to support
other environments" for most of the cases.

On Thu, 24 Jun 2021, 23:06 Sage Weil,  wrote:

> On Tue, Jun 22, 2021 at 1:25 PM Stefan Kooman  wrote:
> > On 6/21/21 6:19 PM, Nico Schottelius wrote:
> > > And while we are at claiming "on a lot more platforms", you are at the
> > > same time EXCLUDING a lot of platforms by saying "Linux based
> > > container" (remember Ceph on FreeBSD? [0]).
> >
> > Indeed, and that is a more fundamental question: how easy it is to make
> > Ceph a first-class citizen on non linux platforms. Was that ever a
> > (design) goal? But then again, if you would be able to port docker
> > natively to say OpenBSD, you should be able to run Ceph on it as well.
>
> Thank you for bringing this up.  This is in fact a key reason why the
> orchestration abstraction works the way it does--to allow other
> runtime environments to be supported (FreeBSD!
> sysvinit/Devuan/whatever for systemd haters!) while ALSO allowing an
> integrated, user-friendly experience in which users workflow for
> adding/removing hosts, replacing failed OSDs, managing services (MDSs,
> RGWs, load balancers, etc) can be consistent across all platforms.
> For 10+ years we basically said "out of scope" to these pesky
> deployment details and left this job to Puppet, Chef, Ansible,
> ceph-deploy, rook, etc., but the result of that strategy was pretty
> clear: ceph was hard to use and the user experience dismal when
> compared to an integrated product from any half-decent enterprise
> storage company, or products like Martin's that capitalize on core
> ceph's bad UX.
>
> The question isn't whether we support other environments, but how.  As
> I mentioned in one of my first messages, we can either (1) generalize
> cephadm to work in other environments (break the current
> systemd+container requirement), or (2) add another orchestrator
> backend that supports a new environment.  I don't have any well-formed
> opinion here.  There is a lot of pretty generic "orchestration" logic
> in cephadm right now that isn't related to systemd or containers that
> could either be pulled out of cephadm into the mgr/ochestrator layer
> or a library.  Or an independent, fresh orch backend implementation
> could opt for a very different approach or set of opinions.
>
> Either way, my assumption has been that these other environments would
> probably not be docker|podman-based.  In the case of FreeBSD we'd
> probably want to use jails or whatever.  But anything is possible.
>
> s
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph buckets

[Rok Jaklič]

Which mode is that and where can I set it?

This one described in https://docs.ceph.com/en/latest/radosgw/multitenancy/
?

On Tue, Jun 8, 2021 at 2:24 PM Janne Johansson  wrote:

> Den tis 8 juni 2021 kl 12:38 skrev Rok Jaklič :
> > Hi,
> > I try to create buckets through rgw in following order:
> > - *bucket1* with *user1* with *access_key1* and *secret_key1*
> > - *bucket1* with *user2* with *access_key2* and *secret_key2*
> >
> > when I try to create a second bucket1 with user2 I get *Error response
> code
> > BucketAlreadyExists.*
> >
> > Why? Should not buckets relate only to users? Is this by design and is
> > there any particular reason that it follows this concept?
>
> If you want to use
> http(s)://bucketname.domain.name/ to access buckets it certainly helps if
> they
> are uniquely named, no?
>
> Apart from that, there is a mode for RGW with tenant/bucketname where
> overlapping/duplicate bucket names are possible, but of course you
> will not get simple DNS-names for published content in that case.
>
> --
> May the most significant bit of your life be positive.
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] ceph buckets

[Rok Jaklič]

Hi,

I try to create buckets through rgw in following order:
- *bucket1* with *user1* with *access_key1* and *secret_key1*
- *bucket1* with *user2* with *access_key2* and *secret_key2*

when I try to create a second bucket1 with user2 I get *Error response code
BucketAlreadyExists.*

Why? Should not buckets relate only to users? Is this by design and is
there any particular reason that it follows this concept?

Regards,
Rok
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Rok Jaklič]

In this giga, tera byte times all this dependency hell can now be avoided
with some static linking. For example, we do use statically linked mysql
binaries and it saved us numerous times. https://youtu.be/5PmHRSeA2c8?t=490

Rok

On Wed, Jun 2, 2021 at 9:57 PM Harry G. Coin  wrote:

>
> On 6/2/21 2:28 PM, Phil Regnauld wrote:
> > Dave Hall (kdhall) writes:
> >> But the developers aren't out in the field with their deployments
> >> when something weird impacts a cluster and the standard approaches don't
> >> resolve it.  And let's face it:  Ceph is a marvelously robust solution
> for
> >> large scale storage, but it is also an amazingly intricate matrix of
> >> layered interdependent processes, and you haven't got all of the bugs
> >> worked out yet.
> >   I think you hit a very important point here: the concern with
> >   containerized deployments is that they may be a barrier to
> >   efficient troubleshooting and bug reporting by traditional methods
> >   (strace et al) -- unless a well documented debugging and analysis
> >   toolset/methodolgy is provided.
> >
> >   Paradoxically, containerized deployments certainly sound like
> they'd
> >   free up lots of cycles from the developer side of things (no more
> >   building packages for N distributions as was pointed out, easier
> >   upgrade and regression testing), but it might make it more
> difficult
> >   initially for the community to contribute (well, at least for us
> >   dinosaurs that aren't born with docker brains).
> >
> >   Cheers,
> >   Phil
>
>
> I think there's great value in ceph devs doing QA and testing docker
> images, releasing them as a 'known good thing'.  Why? Doing that avoids
> dependency hell inducing fragility-- fragility which I've experienced in
> other multi-host / multi-master packages.  Wherein one distro's
> maintainer decides some new rev ought be pushed out as 'security update'
> while another distro's maintainer decides it's a feature change, another
> calls it a backport, etc.  There's no way to QA 'upgrades' across so
> many grains of shifting sand.
>
> While the devs and the rest of the bleeding-edge folks should enjoy the
> benefits that come with tolerating and managing dependency hell, having
> the orchestrator upgrade in a known good sequence from a known base to a
> known release reduces fragility.
>
> Thanks for ceph!
>
> Harry
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Rok Jaklič]

I agree, simplifying "deployment" by adding another layer of complexity
does bring much more problems and hard times when something goes wrong in
the runtime. Few additional steps at "install phase" and better
understanding of underlying architecture, commands, whatever  ... have much
more pros than cons at the end. Even in the most simplified installation,
deployment, systems are complex by itself enough so I have no idea why
under the recommended installation method
https://docs.ceph.com/en/latest/cephadm/install/ there are lvm, systemd,
docker or podman.

I suggest that much effort should be put into better and more
understandable documentation.

Rok


On Wed, Jun 2, 2021 at 11:36 AM Matthew Vernon  wrote:

> Hi,
>
> In the discussion after the Ceph Month talks yesterday, there was a bit
> of chat about cephadm / containers / packages. IIRC, Sage observed that
> a common reason in the recent user survey for not using cephadm was that
> it only worked on containerised deployments. I think he then went on to
> say that he hadn't heard any compelling reasons why not to use
> containers, and suggested that resistance was essentially a user
> education question[0].
>
> I'd like to suggest, briefly, that:
>
> * containerised deployments are more complex to manage, and this is not
> simply a matter of familiarity
> * reducing the complexity of systems makes admins' lives easier
> * the trade-off of the pros and cons of containers vs packages is not
> obvious, and will depend on deployment needs
> * Ceph users will benefit from both approaches being supported into the
> future
>
> We make extensive use of containers at Sanger, particularly for
> scientific workflows, and also for bundling some web apps (e.g.
> Grafana). We've also looked at a number of container runtimes (Docker,
> singularity, charliecloud). They do have advantages - it's easy to
> distribute a complex userland in a way that will run on (almost) any
> target distribution; rapid "cloud" deployment; some separation (via
> namespaces) of network/users/processes.
>
> For what I think of as a 'boring' Ceph deploy (i.e. install on a set of
> dedicated hardware and then run for a long time), I'm not sure any of
> these benefits are particularly relevant and/or compelling - Ceph
> upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud
> Archive) provide .debs of a couple of different Ceph releases per Ubuntu
> LTS - meaning we can easily separate out OS upgrade from Ceph upgrade.
> And upgrading the Ceph packages _doesn't_ restart the daemons[1],
> meaning that we maintain control over restart order during an upgrade.
> And while we might briefly install packages from a PPA or similar to
> test a bugfix, we roll those (test-)cluster-wide, rather than trying to
> run a mixed set of versions on a single cluster - and I understand this
> single-version approach is best practice.
>
> Deployment via containers does bring complexity; some examples we've
> found at Sanger (not all Ceph-related, which we run from packages):
>
> * you now have 2 process supervision points - dockerd and systemd
> * docker updates (via distribution unattended-upgrades) have an
> unfortunate habit of rudely restarting everything
> * docker squats on a chunk of RFC 1918 space (and telling it not to can
> be a bore), which coincides with our internal network...
> * there is more friction if you need to look inside containers
> (particularly if you have a lot running on a host and are trying to find
> out what's going on)
> * you typically need to be root to build docker containers (unlike
> packages)
> * we already have package deployment infrastructure (which we'll need
> regardless of deployment choice)
>
> We also currently use systemd overrides to tweak some of the Ceph units
> (e.g. to do some network sanity checks before bringing up an OSD), and
> have some tools to pair OSD / journal / LVM / disk device up; I think
> these would be more fiddly in a containerised deployment. I'd accept
> that fixing these might just be a SMOP[2] on our part.
>
> Now none of this is show-stopping, and I am most definitely not saying
> "don't ship containers". But I think there is added complexity to your
> deployment from going the containers route, and that is not simply a
> "learn how to use containers" learning curve. I do think it is
> reasonable for an admin to want to reduce the complexity of what they're
> dealing with - after all, much of my job is trying to automate or
> simplify the management of complex systems!
>
> I can see from a software maintainer's point of view that just building
> one container and shipping it everywhere is easier than building
> packages for a number of different distributions (one of my other hats
> is a Debian developer, and I have a bunch of machinery for doing this
> sort of thing). But it would be a bit unfortunate if the general thrust
> of "let's make Ceph easier to set up and manage" was somewhat derailed
> with "you must use containers

[ceph-users] time duration of radosgw-admin

[Rok Jaklič]

Hi,

is it normal that radosgw-admin user info --uid=user ... takes around 3s or
more?

Also other radosgw-admin are taking quite a lot of time.

Kind regards,
Rok
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: rebalancing after node more

[Rok Jaklič]

16.2.4

for some when starting ods with systemctl on this "renewed" host, did not
start osds after a while, but when doing it through console manually, it
did.

Thank anyway.

On Thu, 27 May 2021, 16:31 Eugen Block,  wrote:

> Yes, if your pool requires 5 chunks and you only have 5 hosts (with
> failure domain host) your PGs become undersized when a host fails and
> won't recover until the OSDs come back. Which ceph version is this?
>
>
> Zitat von Rok Jaklič :
>
> > For this pool I have set EC 3+2 (so in total I have 5 nodes) which one
> was
> > temporarily removed, but maybe this was the problem?
> >
> > On Thu, May 27, 2021 at 3:51 PM Rok Jaklič  wrote:
> >
> >> Hi, thanks for quick reply
> >>
> >> root@ctplmon1:~# ceph pg dump pgs_brief | grep undersized
> >> dumped pgs_brief
> >> 9.5  active+undersized+degraded   [72,85,54,120,2147483647]
> >> 72   [72,85,54,120,2147483647]  72
> >> 9.6  active+undersized+degraded  [101,47,113,74,2147483647]
> >>  101  [101,47,113,74,2147483647] 101
> >> 9.2  active+undersized+degraded   [86,118,74,2147483647,49]
> >> 86   [86,118,74,2147483647,49]  86
> >> 9.d  active+undersized+degraded   [49,136,83,90,2147483647]
> >> 49   [49,136,83,90,2147483647]  49
> >> 9.f  active+undersized+degraded  [55,103,81,128,2147483647]
> >> 55  [55,103,81,128,2147483647]  55
> >> 9.18 active+undersized+degraded   [115,50,61,89,2147483647]
> >>  115   [115,50,61,89,2147483647] 115
> >> 9.1d active+undersized+degraded   [61,90,31,2147483647,125]
> >> 61   [61,90,31,2147483647,125]  61
> >> 9.10 active+undersized+degraded   [46,2147483647,71,86,122]
> >> 46   [46,2147483647,71,86,122]  46
> >> 9.17 active+undersized+degraded   [60,95,114,2147483647,48]
> >> 60   [60,95,114,2147483647,48]  60
> >> 9.15 active+undersized+degraded  [121,76,30,101,2147483647]
> >>  121  [121,76,30,101,2147483647] 121
> >> root@ctplmon1:~# ceph osd tree
> >> ID   CLASS  WEIGHT TYPE NAME  STATUS  REWEIGHT  PRI-AFF
> >>  -1 764.11981  root default
> >>  -3 152.82378  host ctplosd1
> >>   0hdd5.45798  osd.0down 0  1.0
> >>   1hdd5.45799  osd.1down 0  1.0
> >>   2hdd5.45799  osd.2down 0  1.0
> >>   3hdd5.45799  osd.3down 0  1.0
> >>   4hdd5.45799  osd.4down 0  1.0
> >>   5hdd5.45799  osd.5down 0  1.0
> >>   6hdd5.45799  osd.6down 0  1.0
> >>   7hdd5.45799  osd.7down 0  1.0
> >>   8hdd5.45799  osd.8down 0  1.0
> >>   9hdd5.45799  osd.9down 0  1.0
> >>  10hdd5.45799  osd.10   down 0  1.0
> >>  11hdd5.45799  osd.11   down 0  1.0
> >>  12hdd5.45799  osd.12   down 0  1.0
> >>  13hdd5.45799  osd.13   down 0  1.0
> >>  14hdd5.45799  osd.14   down 0  1.0
> >>  15hdd5.45799  osd.15   down 0  1.0
> >>  16hdd5.45799  osd.16   down 0  1.0
> >>  17hdd5.45799  osd.17   down 0  1.0
> >>  18hdd5.45799  osd.18   down 0  1.0
> >>  19hdd5.45799  osd.19   down 0  1.0
> >>  20hdd5.45799  osd.20   down 0  1.0
> >>  21hdd5.45799  osd.21   down 0  1.0
> >>  22hdd5.45799  osd.22   down 0  1.0
> >>  23hdd5.45799  osd.23   down 0  1.0
> >>  24hdd5.45799  osd.24   down 0  1.0
> >>  25hdd5.45799  osd.25   down 0  1.0
> >>  26hdd5.45799  osd.26   down 0  1.0
> >>  27hdd5.45799  osd.27   down 0  1.0
> >> -11 152.82401  host ctplosd5
> >> 112hdd5.45799  osd.112up   1.0  1.0
> >> 113hdd5.45799 

[ceph-users] Re: rebalancing after node more

[Rok Jaklič]

For this pool I have set EC 3+2 (so in total I have 5 nodes) which one was
temporarily removed, but maybe this was the problem?

On Thu, May 27, 2021 at 3:51 PM Rok Jaklič  wrote:

> Hi, thanks for quick reply
>
> root@ctplmon1:~# ceph pg dump pgs_brief | grep undersized
> dumped pgs_brief
> 9.5  active+undersized+degraded   [72,85,54,120,2147483647]
> 72   [72,85,54,120,2147483647]  72
> 9.6  active+undersized+degraded  [101,47,113,74,2147483647]
>  101  [101,47,113,74,2147483647] 101
> 9.2  active+undersized+degraded   [86,118,74,2147483647,49]
> 86   [86,118,74,2147483647,49]  86
> 9.d  active+undersized+degraded   [49,136,83,90,2147483647]
> 49   [49,136,83,90,2147483647]  49
> 9.f  active+undersized+degraded  [55,103,81,128,2147483647]
> 55  [55,103,81,128,2147483647]  55
> 9.18 active+undersized+degraded   [115,50,61,89,2147483647]
>  115   [115,50,61,89,2147483647] 115
> 9.1d active+undersized+degraded   [61,90,31,2147483647,125]
> 61   [61,90,31,2147483647,125]  61
> 9.10 active+undersized+degraded   [46,2147483647,71,86,122]
> 46   [46,2147483647,71,86,122]  46
> 9.17 active+undersized+degraded   [60,95,114,2147483647,48]
> 60   [60,95,114,2147483647,48]  60
> 9.15 active+undersized+degraded  [121,76,30,101,2147483647]
>  121  [121,76,30,101,2147483647] 121
> root@ctplmon1:~# ceph osd tree
> ID   CLASS  WEIGHT TYPE NAME  STATUS  REWEIGHT  PRI-AFF
>  -1 764.11981  root default
>  -3 152.82378  host ctplosd1
>   0hdd5.45798  osd.0down 0  1.0
>   1hdd5.45799  osd.1down 0  1.0
>   2hdd5.45799  osd.2down 0  1.0
>   3hdd5.45799  osd.3down 0  1.0
>   4hdd5.45799  osd.4down 0  1.0
>   5hdd5.45799  osd.5down 0  1.0
>   6hdd5.45799  osd.6down 0  1.0
>   7hdd5.45799  osd.7down 0  1.0
>   8hdd5.45799  osd.8down 0  1.0
>   9hdd5.45799  osd.9down 0  1.0
>  10hdd5.45799  osd.10   down 0  1.0
>  11hdd5.45799  osd.11   down 0  1.0
>  12hdd5.45799  osd.12   down 0  1.0
>  13hdd5.45799  osd.13   down 0  1.0
>  14hdd5.45799  osd.14   down 0  1.0
>  15hdd5.45799  osd.15   down 0  1.0
>  16hdd5.45799  osd.16   down 0  1.0
>  17hdd5.45799  osd.17   down 0  1.0
>  18hdd5.45799  osd.18   down 0  1.0
>  19hdd5.45799  osd.19   down 0  1.0
>  20hdd5.45799  osd.20   down 0  1.0
>  21hdd5.45799  osd.21   down 0  1.0
>  22hdd5.45799  osd.22   down 0  1.0
>  23hdd5.45799  osd.23   down 0  1.0
>  24hdd5.45799  osd.24   down 0  1.0
>  25hdd5.45799  osd.25   down 0  1.0
>  26hdd5.45799  osd.26   down 0  1.0
>  27hdd5.45799  osd.27   down 0  1.0
> -11 152.82401  host ctplosd5
> 112hdd5.45799  osd.112up   1.0  1.0
> 113hdd5.45799  osd.113up   1.0  1.0
> 114hdd5.45799  osd.114up   1.0  1.0
> 115hdd5.45799  osd.115up   1.0  1.0
> 116hdd5.45799  osd.116up   1.0  1.0
> 117hdd5.45799  osd.117up   1.0  1.0
> 118hdd5.45799  osd.118up   1.0  1.0
> 119hdd5.45799  osd.119up   1.0  1.0
> 120hdd5.45799  osd.120up   1.0  1.0
> 121hdd5.45799  osd.121up   1.0  1.0
> 122hdd5.45799  osd.122up   1.0  1.0
> 123hdd5.45799  osd.123up   1.0  1.0
> 124hdd5.45799  osd.124up   1.0  1.0
> 125hdd5.45799  osd.125up   1.0  1.0
> 126hdd5.45799  osd.126up   1.0  1.0
> 127hdd5.45799  osd.127up   1.0  1.0
> 128hd

[ceph-users] Re: rebalancing after node more

[Rok Jaklič]

hdd5.45799  osd.96 up   1.0  1.0
 97hdd5.45799  osd.97 up   1.0  1.0
 98hdd5.45799  osd.98 up   1.0  1.0
 99hdd5.45799  osd.99 up   1.0  1.0
100hdd5.45799  osd.100up   1.0  1.0
101hdd5.45799  osd.101up   1.0  1.0
102hdd5.45799  osd.102up   1.0  1.0
103hdd5.45799  osd.103up   1.0  1.0
104hdd5.45799  osd.104up   1.0  1.0
105hdd5.45799  osd.105up   1.0  1.0
106hdd5.45799  osd.106up   1.0  1.0
107hdd5.45799  osd.107up   1.0  1.0
108hdd5.45799  osd.108up   1.0  1.0
109hdd5.45799  osd.109up   1.0  1.0
110hdd5.45799  osd.110up   1.0  1.0
111hdd5.45799  osd.111up   1.0  1.0
root@ctplmon1:~# ceph osd pool ls detail
pool 9 'default.rgw.buckets.data' erasure profile ec-32-profile size 5
min_size 4 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32
autoscale_mode on last_change 128267 lfor 0/127784/127779 flags
hashpspool,ec_overwrites stripe_width 12288 application rgw



The affected pool is pool number 9 and host is ctplosd1. This is the host I
removed at the first place (to reinstall OS) and now I added this host back
to the cluster, but osds on this host cannot be brought back to up state
for some reason, even though osd processes are running on the host.

Kind regards,
Rok





On Thu, May 27, 2021 at 3:32 PM Eugen Block  wrote:

> Hi,
>
> this sounds like your crush rule(s) for one or more pools can't place
> the PGs because the host is missing. Please share
>
> ceph pg dump pgs_brief | grep undersized
> ceph osd tree
> ceph osd pool ls detail
>
> and the crush rule(s) for the affected pool(s).
>
>
> Zitat von Rok Jaklič :
>
> > Hi,
> >
> > I have removed one node, but now ceph seems to stuck in:
> > Degraded data redundancy: 67/2393 objects degraded (2.800%), 12 pgs
> > degraded, 12 pgs undersized
> >
> > How to "force" rebalancing? Or should I just wait a little bit more?
> >
> > Kind regards,
> > rok
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] rebalancing after node more

[Rok Jaklič]

Hi,

I have removed one node, but now ceph seems to stuck in:
Degraded data redundancy: 67/2393 objects degraded (2.800%), 12 pgs
degraded, 12 pgs undersized

How to "force" rebalancing? Or should I just wait a little bit more?

Kind regards,
rok
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph osd df size shows wrong, smaller number

[Rok Jaklič]

Ahh, thx for noticing that. Its the same for 56.

We've figured out that this maybe due to the fact that this manual method
is for filestore
https://docs.ceph.com/en/latest/install/manual-deployment/#long-form only.
There isnt any manual method for bluestore.





On Fri, May 21, 2021 at 11:00 AM Janne Johansson 
wrote:

> Den fre 21 maj 2021 kl 10:49 skrev Rok Jaklič :
> > It shows
> > sdb8:16   0   5.5T  0 disk /var/lib/ceph/osd/ceph-56
>
> That one says osd-56, you asked about why osd 85 was small in ceph osd df
>
>
> >> Den fre 21 maj 2021 kl 09:41 skrev Rok Jaklič :
> >> > why would ceph osd df show in SIZE field smaller number than there is:
> >> >  85hdd  0.8   1.0  100 GiB   96 GiB   95 GiB   289 KiB
>  952
> >> > MiB  4.3 GiB  95.68  3.37   10  up
> >> >
> >> > instead of 100GiB there should be 5.5TiB.
> >>
> >> What does "lsblk" say about the size of the disk/partition where osd 85
> runs?
> >>
> >>
> >> --
> >> May the most significant bit of your life be positive.
>
>
>
> --
> May the most significant bit of your life be positive.
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph osd df size shows wrong, smaller number

[Rok Jaklič]

It shows

sdb8:16   0   5.5T  0 disk /var/lib/ceph/osd/ceph-56

when I do ll on that dir it says
[root@ctplosd8 ~]# ll /var/lib/ceph/osd/ceph-56
total 552
-rw--- 1 root root9 May 21 10:40 bfm_blocks
-rw--- 1 root root4 May 21 10:40 bfm_blocks_per_key
-rw--- 1 root root5 May 21 10:40 bfm_bytes_per_block
-rw--- 1 root root   13 May 21 10:40 bfm_size
-rw-r--r-- 1 root root 107374182400 May 21 10:41 block
-rw--- 1 root root2 May 21 10:40 bluefs
-rw--- 1 root root   37 May 21 10:41 ceph_fsid
-rw-r--r-- 1 root root   37 May 21 10:40 fsid
-rw--- 1 root root8 May 21 10:40 kv_backend
-rw--- 1 root root   21 May 21 10:41 magic
-rw--- 1 root root4 May 21 10:41 mkfs_done
-rw--- 1 root root6 May 21 10:41 ready
-rw--- 1 root root   10 May 21 10:40 type
-rw--- 1 root root3 May 21 10:41 whoami

...

I created osds manually with following script
UUID=$(uuidgen)
ID=$(ceph osd new $UUID)

mkdir /var/lib/ceph/osd/ceph-$ID
mkfs.xfs /dev/$disk
mount /dev/$disk /var/lib/ceph/osd/ceph-$ID/
ceph-osd -i $ID --mkfs --osd-uuid $UUID --data /dev/sdb
chown -R ceph:ceph /var/lib/ceph/osd/ceph-$ID/

---

and there 100G block file resides.


On Fri, May 21, 2021 at 9:59 AM Janne Johansson  wrote:

> Den fre 21 maj 2021 kl 09:41 skrev Rok Jaklič :
> > why would ceph osd df show in SIZE field smaller number than there is:
> >  85hdd  0.8   1.0  100 GiB   96 GiB   95 GiB   289 KiB   952
> > MiB  4.3 GiB  95.68  3.37   10  up
> >
> > instead of 100GiB there should be 5.5TiB.
>
> What does "lsblk" say about the size of the disk/partition where osd 85
> runs?
>
>
> --
> May the most significant bit of your life be positive.
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] ceph osd df size shows wrong, smaller number

[Rok Jaklič]

Hi,

why would ceph osd df show in SIZE field smaller number than there is:
 85hdd  0.8   1.0  100 GiB   96 GiB   95 GiB   289 KiB   952
MiB  4.3 GiB  95.68  3.37   10  up

instead of 100GiB there should be 5.5TiB.

Kind regards,
Rok
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Configuring an S3 gateway

[Rok Jaklič]

I agree. Documentation here is pretty vague. systemd services for osds on
ubuntu 20.04 and ceph pacific version 16.2.1 does not work either, so I
have to run it manually with

/usr/bin/ceph-osd -f --cluster ceph --id some-number --setuser ceph
--setgroup ceph

I think it would be much better if documentation would focus on how to run
it manually without systemd help, because distributions are not all "the
same" even with systemd in place.

Rok

V V čet., 22. apr. 2021 ob 21:29 je oseba Fabrice Bacchella <
fabrice.bacche...@orange.fr> napisala:

>
>
> > Le 22 avr. 2021 à 20:40, Casey Bodley  a écrit :
> >
> > On Thu, Apr 22, 2021 at 2:26 PM Fabrice Bacchella
> > mailto:fabrice.bacche...@orange.fr>>
> wrote:
> >>
> >> I'm trying to configure an S3 gateway with pacific and can't wrap my
> mind around.
> >>
> >> In the configuration file, my configuration is:
> >>
> >> [client.radosgw.fa41]
> >>  rgw_data = /data/ceph/data/radosgw/$cluster.$id
> >>  log_file = /data/ceph/logs/$cluster-radosgw.$id.log
> >>  rgw_frontends = "beast ssl_endpoint=0.0.0.0:443
> ssl_certificate=/data/ceph/conf/ceph.crt
> ssl_private_key=/data/ceph/conf/ceph.key"
> >>
> >> and radosgw ignore it:
> >>
> >> # /usr/bin/radosgw -d --cluster ngceph --name client.fa41 --setuser
> ceph --setgroup ceph
> >
> > the --name should match the ceph.conf section. does '--name
> > client.radosgw.fa41' work?
>
> Indeed it works much better now, thanks.
>
> But I took the argument from the unit:
> ExecStart=/usr/bin/radosgw -f --cluster ${CLUSTER} --name client.%i
> --setuser ceph --setgroup ceph
>
> So to works, the unit to start should be ceph-radosgw@radosgw.fa41.service.
> That's would be a little strange.
>
> And in the dashboard, it's shown as with an ID of radosgw.fa41 in the
> ObjectGatway screen. Isn't that some kind of inconcistency about what is
> the ID for a client ?
>
> Meanwhile, neither
> /usr/bin/radosgw -d --cluster ngceph -i radosgw.fa41 --setuser ceph
> --setgroup ceph
> or
> /usr/bin/radosgw -d --cluster ngceph -i fa41 --setuser ceph --setgroup ceph
>
>
> I've installed ceph-radosgw-16.2.1-0.el8.x86_64 directly from the ceph
> repositories.
>
>
> >
> >> 2021-04-22T20:19:44.362+0200 7fdf51416480  0 deferred set uid:gid to
> 167:167 (ceph:ceph)
> >> 2021-04-22T20:19:44.363+0200 7fdf51416480  0 ceph version 16.2.1
> (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable), process
> radosgw, pid 9780
> >> 2021-04-22T20:19:44.363+0200 7fdf51416480  0 framework: beast
> >> 2021-04-22T20:19:44.363+0200 7fdf51416480  0 framework conf key: port,
> val: 7480
> >> 2021-04-22T20:19:44.363+0200 7fdf51416480  1 radosgw_Main not setting
> numa affinity
> >> 2021-04-22T20:19:45.585+0200 7fdf51416480  0 framework: beast
> >> 2021-04-22T20:19:45.586+0200 7fdf51416480  0 framework conf key:
> ssl_certificate, val: config://rgw/cert/$realm/$zone.crt
> >> 2021-04-22T20:19:45.586+0200 7fdf51416480  0 framework conf key:
> ssl_private_key, val: config://rgw/cert/$realm/$zone.key
> >> 2021-04-22T20:19:45.586+0200 7fdf51416480  0 starting handler: beast
> >> 2021-04-22T20:19:45.592+0200 7fdf51416480  0 WARNING: cannot open
> socket for endpoint=[::]:7480, Address family not supported by protocol
> >> 2021-04-22T20:19:45.627+0200 7fdf51416480  0 set uid:gid to 167:167
> (ceph:ceph)
> >> 2021-04-22T20:19:45.811+0200 7fdf51416480  1 mgrc
> service_daemon_register rgw.134130 metadata
> {arch=x86_64,ceph_release=pacific,ceph_version=ceph version 16.2.1
> (afb9061ab4117f798c858c741efa6390e48ccf10) pacific
> (stable),ceph_version_short=16.2.1,cpu=Intel Core Processor (Haswell, no
> TSX),distro=centos,distro_description=CentOS Linux
> 8,distro_version=8,frontend_config#0=beast
> port=7480,frontend_type#0=beast,hostname=fa41,id=fa41,kernel_description=#1
> SMP Thu Apr 8 19:01:30 UTC
> 2021,kernel_version=4.18.0-240.22.1.el8_3.x86_64,mem_swap_kb=0,mem_total_kb=16211232,num_handles=1,os=Linux,pid=9780,zone_id=2dc75a54-8c59-42bc-98a8-35542fdc4e52,zone_name=default,zonegroup_id=d11b8d14-7608-4b1d-a548-09b5dd813a7a,zonegroup_name=default}
> >>
> >> I don't get what I'm missing. Is there any typo in the configuration
> that I'm missing ?
> >>
> >> I've verified using strace, and it reads the expected configuration
> file.
> >> ___
> >> ceph-users mailing list -- ceph-users@ceph.io
> >> To unsubscribe send an email to ceph-users-le...@ceph.io
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Ceph Object Gateway setup/tutorial

[Rok Jaklič]

Hi,

installation of cluster/osds went "by the book" https://docs.ceph.com/, but
now I want to setup Ceph Object Gateway, but documentation on
https://docs.ceph.com/en/latest/radosgw/ seems to lack information about
what and where to restart for example when setting [client.rgw.gateway-node1]
in /etc/ceph/ceph.conf. Also where should we set this? In cephadm shell or
on the host ...?

Is there some tutorial howto setup gateway from the beginning?

Kind regards,
Rok
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] File listing with browser

[Rok Jaklič]

Hi,

I installed ceph object gateway and I have put one test object onto
storage. I can see it with rados -p mytest ls

How do I setup ceph that users can access (download,upload) files to this
pool?

Kind regards,

Rok
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io