[ceph-users] Re: Deep-scrub much slower than HDD speed

[Janne Johansson]

Den fre 28 apr. 2023 kl 14:51 skrev Niklas Hambüchen :
>
> Hi all,
>
> > Scrubs only read data that does exist in ceph as it exists, not every 
> > sector of the drive, written or not.
>
> Thanks, this does explain it.
>
> I just discovered:
>
> ZFS had this problem in the past:
>
> * 
> https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSNonlinearScrubs?showcomments#comments

That one talks about resilvering, which is not the same as neither ZFS
scrubs nor ceph scrubs.
Resilvering is copying/moving/rewriting data, which scrubs are not.

-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: architecture help (iscsi, rbd, backups?)

[Alex Gorbachev]

Hi Angelo,

You can always use Samba to serve shares, it works well with AD, if that is
needed.  You may want to benchmark your prototypes in an as close to
production setting as possible.

--
Alex Gorbachev
ISS Storcium
iss-integration.com



On Sat, Apr 29, 2023 at 10:58 PM Angelo Hongens  wrote:

>
> Thanks Alex, interesting perspectives.
>
> I already thought about proxmox as well, and that would also work quite
> nicely. I think that would be the most performant option to put VM's on
> RBD.
>
> But my entire goal was to run SMB servers on top of that hypervisor
> layer, to serve SMB shares to Windows.
>
> So I think Bailey's suggestion makes more sense, to use CephFS with
> Linux SMB gateways, which cuts out a layer in between, greatly improving
> performance.
>
> That also has the benefit of being able to use a single 1PB CephFS
> filesystem served by multiple SMB gateways instead of my initial plan of
> having like 10x100TB windows SMB file servers (I would not dare have a
> single 1PB windows VM with an NTFS disk)
>
>
>
> Angelo.
>
> On 27/04/2023 20:05, Alex Gorbachev wrote:
> > Hi Angelo,
> >
> > Just some thoughts to consider from our experience with similar setups:
> >
> > 1. Use Proxmox instead of VMWare, or anything KVM based.  These VMs can
> > consume Ceph directly, and provide the same level of service (some may
> > say better) for live ,migration, hyperconvergence etc.  Then you run
> > Windows VMs in KVM, bring RBD storage to them as virtual disks and share
> > out as needed.
> >
> > 2. Use NFS - all modern Windows  OSs support it.  You can use any NFS
> > gateway you like, or set up your own machine or cluster (which is what
> > we did with Storcium) and export your storage as needed.
> >
> > 3. If you must use VMWare, you can present datastores via NFS as well,
> > this has a lot of indirection but is easier to manage.
> >
> > --
> > Alex Gorbachev
> > ISS Storcium
> > https://www.iss-integration.com 
> >
> >
> >
> > On Thu, Apr 27, 2023 at 5:06 PM Angelo Höngens  > > wrote:
> >
> > Hey guys and girls,
> >
> > I'm working on a project to build storage for one of our departments,
> > and I want to ask you guys and girls for input on the high-level
> > overview part. It's a long one, I hope you read along and comment.
> >
> > SUMMARY
> >
> > I made a plan last year to build a 'storage solution' including ceph
> > and some windows VM's to expose the data over SMB to clients. A year
> > later I finally have the hardware, built a ceph cluster, and I'm
> doing
> > tests. Ceph itself runs great, but when I wanted to start exposing
> the
> > data using iscsi to our VMware farm, I ran into some issues. I know
> > the iscsi gateways will introduce some new performance bottlenecks,
> > but I'm seeing really slow performance, still working on that.
> >
> > But then I ran into the warning on the iscsi gateway page: "The iSCSI
> > gateway is in maintenance as of November 2022. This means that it is
> > no longer in active development and will not be updated to add new
> > features.". Wait, what? Why!? What does this mean? Does this mean
> that
> > iSCSI is now 'feature complete' and will still be supported the next
> 5
> > years, or will it be deprecated in the future? I tried searching, but
> > couldn't find any info on the decision and the roadmap.
> >
> > My goal is to build a future-proof setup, and using deprecated
> > components should not be part of that of course.
> >
> > If the iscsi gateway will still be supported the next few years and I
> > can iron out the performance issues, I can still go on with my
> > original plan. If not, I have to go back to the drawing board. And
> > maybe you guys would advise me to take another route anyway.
> >
> > GOALS
> >
> > My goals/considerations are:
> >
> > - we want >1PB of storage capacity for cheap (on a tight budget) for
> > research data. Most of it is 'store once, read sometimes'. <1% of the
> > data is 'hot'.
> > - focus is on capacity, but it would be nice to have > 200MB/s of
> > sequential write/read performance and not 'totally suck' on random
> > i/o. Yes, not very well quantified, but ah. Sequential writes are
> most
> > important.
> > - end users all run Windows computers (mostly VDI's) and a lot of
> > applications require SMB shares.
> > - security is a big thing, we want really tight ACL's, specific
> > monitoring agents, etc.
> > - our data is incredibly important to us, we still want the 3-2-1
> > backup rule. Primary storage solution, a second storage solution in a
> > different place, and some of the data that is not reproducible is
> also
> > written to tape. We also want to be protected from ransomware or user
> > errors (so no direct replication to the second storage).
> > - I like open source, 

[ceph-users] Re: architecture help (iscsi, rbd, backups?)

[Angelo Hongens]

Thanks Alex, interesting perspectives.

I already thought about proxmox as well, and that would also work quite 
nicely. I think that would be the most performant option to put VM's on 
RBD.


But my entire goal was to run SMB servers on top of that hypervisor 
layer, to serve SMB shares to Windows.


So I think Bailey's suggestion makes more sense, to use CephFS with 
Linux SMB gateways, which cuts out a layer in between, greatly improving 
performance.


That also has the benefit of being able to use a single 1PB CephFS 
filesystem served by multiple SMB gateways instead of my initial plan of 
having like 10x100TB windows SMB file servers (I would not dare have a 
single 1PB windows VM with an NTFS disk)




Angelo.

On 27/04/2023 20:05, Alex Gorbachev wrote:

Hi Angelo,

Just some thoughts to consider from our experience with similar setups:

1. Use Proxmox instead of VMWare, or anything KVM based.  These VMs can 
consume Ceph directly, and provide the same level of service (some may 
say better) for live ,migration, hyperconvergence etc.  Then you run 
Windows VMs in KVM, bring RBD storage to them as virtual disks and share 
out as needed.


2. Use NFS - all modern Windows  OSs support it.  You can use any NFS 
gateway you like, or set up your own machine or cluster (which is what 
we did with Storcium) and export your storage as needed.


3. If you must use VMWare, you can present datastores via NFS as well, 
this has a lot of indirection but is easier to manage.


--
Alex Gorbachev
ISS Storcium
https://www.iss-integration.com 



On Thu, Apr 27, 2023 at 5:06 PM Angelo Höngens > wrote:


Hey guys and girls,

I'm working on a project to build storage for one of our departments,
and I want to ask you guys and girls for input on the high-level
overview part. It's a long one, I hope you read along and comment.

SUMMARY

I made a plan last year to build a 'storage solution' including ceph
and some windows VM's to expose the data over SMB to clients. A year
later I finally have the hardware, built a ceph cluster, and I'm doing
tests. Ceph itself runs great, but when I wanted to start exposing the
data using iscsi to our VMware farm, I ran into some issues. I know
the iscsi gateways will introduce some new performance bottlenecks,
but I'm seeing really slow performance, still working on that.

But then I ran into the warning on the iscsi gateway page: "The iSCSI
gateway is in maintenance as of November 2022. This means that it is
no longer in active development and will not be updated to add new
features.". Wait, what? Why!? What does this mean? Does this mean that
iSCSI is now 'feature complete' and will still be supported the next 5
years, or will it be deprecated in the future? I tried searching, but
couldn't find any info on the decision and the roadmap.

My goal is to build a future-proof setup, and using deprecated
components should not be part of that of course.

If the iscsi gateway will still be supported the next few years and I
can iron out the performance issues, I can still go on with my
original plan. If not, I have to go back to the drawing board. And
maybe you guys would advise me to take another route anyway.

GOALS

My goals/considerations are:

- we want >1PB of storage capacity for cheap (on a tight budget) for
research data. Most of it is 'store once, read sometimes'. <1% of the
data is 'hot'.
- focus is on capacity, but it would be nice to have > 200MB/s of
sequential write/read performance and not 'totally suck' on random
i/o. Yes, not very well quantified, but ah. Sequential writes are most
important.
- end users all run Windows computers (mostly VDI's) and a lot of
applications require SMB shares.
- security is a big thing, we want really tight ACL's, specific
monitoring agents, etc.
- our data is incredibly important to us, we still want the 3-2-1
backup rule. Primary storage solution, a second storage solution in a
different place, and some of the data that is not reproducible is also
written to tape. We also want to be protected from ransomware or user
errors (so no direct replication to the second storage).
- I like open source, reliability, no fork-lift upgrades, no vendor
lock-in, blah, well, I'm on the ceph list here, no need to convince
you guys ;)
- We're hiring a commercial company to do ceph maintenance and support
for when I'm on leave or leaving the company, but they won't support
clients, backup software, etc, so I want something as simple as
possible. We do have multiple Windows/VMware admins, but no other real
linux guru's.

THE INITIAL PLAN

Given these considerations, I ordered two identical clusters, each
consisting of 3 monitor nodes and 8 osd nodes, Each osd node has 2
ssd's and 10 

[ceph-users] Re: architecture help (iscsi, rbd, backups?)

[Angelo Hongens]

Bailey,

Thanks for your extensive reply, you got me down the wormhole of CephFS 
and SMB (and looking at a lot of 45drives videos and knowledge base, 
Houston dashboard, reading up on CTDB, etc), and this is a really 
interesting option as well! Thanks for the write-up.



By the way, are you using the RBD driver in Windows in production with 
your customers?


The binaries are still called beta, and last time I tried it in a proof 
of concept setup (a while back), it would never connect and always crash 
out on me. After reporting an issue, I did not get a response for almost 
three months before a dev responded that it was an unsupported ipv6 
issue. Not a problem, and all very understandable, it's open source 
software written mostly by volunteers, but I got a bit cautious about 
deploying this to production ;)


Angelo.





On 27/04/2023 18:20, Bailey Allison wrote:

Hey Angelo,

Just to make sure I'm understanding correctly, the main idea for the use
case is to be able to present Ceph storage to windows clients as SMB?

If so, you can absolutely use CephFS to get that done. This is something we
do all the time with our cluster configurations, if we're looking to present
ceph storage to windows clients for the use case of a file server is our
standard choice, and to your point of security/ACLs we can make use of
joining the samba server that to an existing active directory, and then
assigning permissions through Windows.

I will provide a high level overview of an average setup to hopefully
explain it better, and of course if you have any questions please let me
know. I understand that this is way different of a setup of what you
currently have planned, but it's a different choice that could prove useful
in your case.

Essentially how it works is we have ceph cluster with CephFS configured, of
which we map CephFS kernel mounts onto some gateway nodes, at which point we
expose to clients via CTDB with SMB shares (CTDB for high availability).

i.e

ceph cluster > ceph fs > map cephfs kernel mount on linux client > create
smb share on top of cephfs kernel mount > connect to samba share with
windows clients.

The SMB gateway nodes hosting samba also can be joined to an Active
Directory to allow setting Windows ACL permissions to allow more in depth
control of ACLs.

Also I will say +1 for the RBD driver on Windows, something we also make use
of a lot and have a lot of success with.

Again, please let me know if you need any insight or clarification, or have
any further questions. Hope this is of assistance.

Regards,

Bailey

-Original Message-

From: Angelo Höngens 
Sent: April 27, 2023 6:06 PM
To: ceph-users@ceph.io
Subject: [ceph-users] architecture help (iscsi, rbd, backups?)

Hey guys and girls,

I'm working on a project to build storage for one of our departments, and I

want to ask you guys and girls for input on the high-level overview part.
It's a long one, I hope you read along and comment.


SUMMARY

I made a plan last year to build a 'storage solution' including ceph and

some windows VM's to expose the data over SMB to clients. A year later I
finally have the hardware, built a ceph cluster, and I'm doing tests. Ceph
itself runs great, but when I wanted to start exposing the data using iscsi
to our VMware farm, I ran into some issues. I know the iscsi gateways will
introduce some new performance bottlenecks, but I'm seeing really slow
performance, still working on that.


But then I ran into the warning on the iscsi gateway page: "The iSCSI

gateway is in maintenance as of November 2022. This means that it is no
longer in active development and will not be updated to add new features.".
Wait, what? Why!? What does this mean? Does this mean that iSCSI is now
'feature complete' and will still be supported the next 5 years, or will it
be deprecated in the future? I tried searching, but couldn't find any info
on the decision and the roadmap.


My goal is to build a future-proof setup, and using deprecated components

should not be part of that of course.


If the iscsi gateway will still be supported the next few years and I can

iron out the performance issues, I can still go on with my original plan. If
not, I have to go back to the drawing board. And maybe you guys would advise
me to take another route anyway.


GOALS

My goals/considerations are:

- we want >1PB of storage capacity for cheap (on a tight budget) for

research data. Most of it is 'store once, read sometimes'. <1% of the data
is 'hot'.

- focus is on capacity, but it would be nice to have > 200MB/s of

sequential write/read performance and not 'totally suck' on random i/o. Yes,
not very well quantified, but ah. Sequential writes are most important.

- end users all run Windows computers (mostly VDI's) and a lot of

applications require SMB shares.

- security is a big thing, we want really tight ACL's, specific monitoring

agents, etc.

- our data is incredibly important to us, we still want the 3-2-1 backup

rule. 

[ceph-users] Re: Help needed to configure erasure coding LRC plugin

[Michel Jouvin]

Hi,

No... our current setup is 3 datacenters with the same configuration, 
i.e. 1 mon/mgr + 4 OSD servers with 16 OSDs each. Thus the total of 12 
OSDs servers. As with LRC plugin, k+m must be a multiple of l, I found 
that k=9/m=66/l=5 with crush-locality=datacenter was achieving my goal 
of being resilient to a datacenter failure. Because I had this, I 
considered that lowering the crush failure domain to osd was not a major 
issue in my case (as it would not be worst than a datacenter failure if 
all the shards are on the same server in a datacenter) and was working 
around the lack of hosts for k=9/m=6 (15 OSDs).


May be it helps, if I give the erasure code profile used:

crush-device-class=hdd
crush-failure-domain=osd
crush-locality=datacenter
crush-root=default
k=9
l=5
m=6
plugin=lrc

The previously mentioned strange number for min_size for the pool 
created with this profile has vanished after Quincy upgrade as this 
parameter is no longer in the CRUH map rule! and the `ceph osd pool get` 
command reports the expected number (10):


-

> ceph osd pool get fink-z1.rgw.buckets.data min_size
min_size: 10


Cheers,

Michel

Le 29/04/2023 à 20:36, Curt a écrit :

Hello,

What is your current setup, 1 server pet data center with 12 osd each? 
What is your current crush rule and LRC crush rule?



On Fri, Apr 28, 2023, 12:29 Michel Jouvin 
 wrote:


Hi,

I think I found a possible cause of my PG down but still
understand why.
As explained in a previous mail, I setup a 15-chunk/OSD EC pool (k=9,
m=6) but I have only 12 OSD servers in the cluster. To workaround the
problem I defined the failure domain as 'osd' with the reasoning
that as
I was using the LRC plugin, I had the warranty that I could loose
a site
without impact, thus the possibility to loose 1 OSD server. Am I
wrong?

Best regards,

Michel

Le 24/04/2023 à 13:24, Michel Jouvin a écrit :
> Hi,
>
> I'm still interesting by getting feedback from those using the LRC
> plugin about the right way to configure it... Last week I upgraded
> from Pacific to Quincy (17.2.6) with cephadm which is doing the
> upgrade host by host, checking if an OSD is ok to stop before
actually
> upgrading it. I had the surprise to see 1 or 2 PGs down at some
points
> in the upgrade (happened not for all OSDs but for every
> site/datacenter). Looking at the details with "ceph health
detail", I
> saw that for these PGs there was 3 OSDs down but I was expecting
the
> pool to be resilient to 6 OSDs down (5 for R/W access) so I'm
> wondering if there is something wrong in our pool configuration
(k=9,
> m=6, l=5).
>
> Cheers,
>
> Michel
>
> Le 06/04/2023 à 08:51, Michel Jouvin a écrit :
>> Hi,
>>
>> Is somebody using LRC plugin ?
>>
>> I came to the conclusion that LRC  k=9, m=3, l=4 is not the
same as
>> jerasure k=9, m=6 in terms of protection against failures and
that I
>> should use k=9, m=6, l=5 to get a level of resilience >= jerasure
>> k=9, m=6. The example in the documentation (k=4, m=2, l=3)
suggests
>> that this LRC configuration gives something better than
jerasure k=4,
>> m=2 as it is resilient to 3 drive failures (but not 4 if I
understood
>> properly). So how many drives can fail in the k=9, m=6, l=5
>> configuration first without loosing RW access and second without
>> loosing data?
>>
>> Another thing that I don't quite understand is that a pool created
>> with this configuration (and failure domain=osd,
locality=datacenter)
>> has a min_size=3 (max_size=18 as expected). It seems wrong to
me, I'd
>> expected something ~10 (depending on answer to the previous
question)...
>>
>> Thanks in advance if somebody could provide some sort of
>> authoritative answer on these 2 questions. Best regards,
>>
>> Michel
>>
>> Le 04/04/2023 à 15:53, Michel Jouvin a écrit :
>>> Answering to myself, I found the reason for 2147483647: it's
>>> documented as a failure to find enough OSD (missing OSDs). And
it is
>>> normal as I selected different hosts for the 15 OSDs but I
have only
>>> 12 hosts!
>>>
>>> I'm still interested by an "expert" to confirm that LRC  k=9,
m=3,
>>> l=4 configuration is equivalent, in terms of redundancy, to a
>>> jerasure configuration with k=9, m=6.
>>>
>>> Michel
>>>
>>> Le 04/04/2023 à 15:26, Michel Jouvin a écrit :
 Hi,

 As discussed in another thread (Crushmap rule for
multi-datacenter
 erasure coding), I'm trying to create an EC pool spanning 3
 datacenters (datacenters are present in the crushmap), with the
 objective to be resilient to 1 DC down, at least keeping the
 readonly access to the pool and if possible the read-write

[ceph-users] Re: Help needed to configure erasure coding LRC plugin

[Curt]

Hello,

What is your current setup, 1 server pet data center with 12 osd each? What
is your current crush rule and LRC crush rule?


On Fri, Apr 28, 2023, 12:29 Michel Jouvin 
wrote:

> Hi,
>
> I think I found a possible cause of my PG down but still understand why.
> As explained in a previous mail, I setup a 15-chunk/OSD EC pool (k=9,
> m=6) but I have only 12 OSD servers in the cluster. To workaround the
> problem I defined the failure domain as 'osd' with the reasoning that as
> I was using the LRC plugin, I had the warranty that I could loose a site
> without impact, thus the possibility to loose 1 OSD server. Am I wrong?
>
> Best regards,
>
> Michel
>
> Le 24/04/2023 à 13:24, Michel Jouvin a écrit :
> > Hi,
> >
> > I'm still interesting by getting feedback from those using the LRC
> > plugin about the right way to configure it... Last week I upgraded
> > from Pacific to Quincy (17.2.6) with cephadm which is doing the
> > upgrade host by host, checking if an OSD is ok to stop before actually
> > upgrading it. I had the surprise to see 1 or 2 PGs down at some points
> > in the upgrade (happened not for all OSDs but for every
> > site/datacenter). Looking at the details with "ceph health detail", I
> > saw that for these PGs there was 3 OSDs down but I was expecting the
> > pool to be resilient to 6 OSDs down (5 for R/W access) so I'm
> > wondering if there is something wrong in our pool configuration (k=9,
> > m=6, l=5).
> >
> > Cheers,
> >
> > Michel
> >
> > Le 06/04/2023 à 08:51, Michel Jouvin a écrit :
> >> Hi,
> >>
> >> Is somebody using LRC plugin ?
> >>
> >> I came to the conclusion that LRC  k=9, m=3, l=4 is not the same as
> >> jerasure k=9, m=6 in terms of protection against failures and that I
> >> should use k=9, m=6, l=5 to get a level of resilience >= jerasure
> >> k=9, m=6. The example in the documentation (k=4, m=2, l=3) suggests
> >> that this LRC configuration gives something better than jerasure k=4,
> >> m=2 as it is resilient to 3 drive failures (but not 4 if I understood
> >> properly). So how many drives can fail in the k=9, m=6, l=5
> >> configuration first without loosing RW access and second without
> >> loosing data?
> >>
> >> Another thing that I don't quite understand is that a pool created
> >> with this configuration (and failure domain=osd, locality=datacenter)
> >> has a min_size=3 (max_size=18 as expected). It seems wrong to me, I'd
> >> expected something ~10 (depending on answer to the previous question)...
> >>
> >> Thanks in advance if somebody could provide some sort of
> >> authoritative answer on these 2 questions. Best regards,
> >>
> >> Michel
> >>
> >> Le 04/04/2023 à 15:53, Michel Jouvin a écrit :
> >>> Answering to myself, I found the reason for 2147483647: it's
> >>> documented as a failure to find enough OSD (missing OSDs). And it is
> >>> normal as I selected different hosts for the 15 OSDs but I have only
> >>> 12 hosts!
> >>>
> >>> I'm still interested by an "expert" to confirm that LRC  k=9, m=3,
> >>> l=4 configuration is equivalent, in terms of redundancy, to a
> >>> jerasure configuration with k=9, m=6.
> >>>
> >>> Michel
> >>>
> >>> Le 04/04/2023 à 15:26, Michel Jouvin a écrit :
>  Hi,
> 
>  As discussed in another thread (Crushmap rule for multi-datacenter
>  erasure coding), I'm trying to create an EC pool spanning 3
>  datacenters (datacenters are present in the crushmap), with the
>  objective to be resilient to 1 DC down, at least keeping the
>  readonly access to the pool and if possible the read-write access,
>  and have a storage efficiency better than 3 replica (let say a
>  storage overhead <= 2).
> 
>  In the discussion, somebody mentioned LRC plugin as a possible
>  jerasure alternative to implement this without tweaking the
>  crushmap rule to implement the 2-step OSD allocation. I looked at
>  the documentation
>  (https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/)
>  but I have some questions if someone has experience/expertise with
>  this LRC plugin.
> 
>  I tried to create a rule for using 5 OSDs per datacenter (15 in
>  total), with 3 (9 in total) being data chunks and others being
>  coding chunks. For this, based of my understanding of examples, I
>  used k=9, m=3, l=4. Is it right? Is this configuration equivalent,
>  in terms of redundancy, to a jerasure configuration with k=9, m=6?
> 
>  The resulting rule, which looks correct to me, is:
> 
>  
> 
>  {
>  "rule_id": 6,
>  "rule_name": "test_lrc_2",
>  "ruleset": 6,
>  "type": 3,
>  "min_size": 3,
>  "max_size": 15,
>  "steps": [
>  {
>  "op": "set_chooseleaf_tries",
>  "num": 5
>  },
>  {
>  "op": "set_choose_tries",
>  "num": 100
>  },
>  {
> 

[ceph-users] Re: cephfs - max snapshot limit?

[Jakob Haufe]

On Fri, 28 Apr 2023 22:46:32 +0530 Milind Changire wrote:

> If a dir doesn't exist at the moment of snapshot creation, then the
> schedule is deactivated for that dir.

Ha! Good catch! As often, I completely forgot about the /volumes prefix...

-- 
ceterum censeo microsoftem esse delendam.


pgpcZRW2lGkZX.pgp
Description: OpenPGP digital signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] RadosGW S3 API Multi-Tenancy

[Brad House]

I'm in the process of exploring if it is worthwhile to add RadosGW to 
our existing ceph cluster.  We've had a few internal requests for 
exposing the S3 API for some of our business units, right now we just 
use the ceph cluster for VM disk image storage via RBD.


Everything looks pretty straight forward until we hit multitenancy. The 
page on multi-tenancy doesn't dive into permission delegation:

https://docs.ceph.com/en/quincy/radosgw/multitenancy/

The end goal I want is to be able to create a single user per tenant 
(Business Unit) which will act as their 'administrator', where they can 
then do basically whatever they want under their tenant sandbox (though 
I don't think we need more advanced cases like creations of roles or 
policies, just create/delete their own users, buckets, objects).  I was 
hopeful this would just work, and I asked on the ceph IRC channel on 
OFTC and was told once I grant a user caps="users=*", they would then be 
allowed to create users *outside* of their own tenant using the Rados 
Admin API and that I should explore IAM roles.


I think it would make sense to add a feature, such as a flag that can be 
set on a user, to ensure they stay in their "sandbox". I'd assume this 
is probably a common use-case.


Anyhow, if its possible to do today using iam roles/policies, then 
great, unfortunately this is my first time looking at this stuff and 
there are some things not immediately obvious.


I saw this online about AWS itself and creating a permissions boundary, 
but that's for allowing creation of roles within a boundary:

https://www.qloudx.com/delegate-aws-iam-user-and-role-creation-without-giving-away-admin-access/

I'm not sure what "Action" is associated with the Rados Admin API create 
user for applying a boundary that the user can only create users with 
the same tenant name.

https://docs.ceph.com/en/quincy/radosgw/adminops/#create-user

Any guidance on this would be extremely helpful.

Thanks!
-Brad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Return code -116 when starting MDS scrub

[Thomas Widhalm]

Hi,

Ok, added "/" and no it works. Thanks.

Saw the "/" in the text I used. Just thought it meant "use either this 
command or that" sorry.


On 29.04.23 13:00, Eugen Block wrote:
Hi, according to this thread [1] it means that the scrub path is not 
existing, but you don’t specify a path at all. Can you retry with „/„? 
Do you see anything with damage ls?


[1] https://www.mail-archive.com/ceph-users@lists.ceph.com/msg56062.html

Zitat von Thomas Widhalm :


Hi,

I followed the steps to repair journal and MDS I found here in the 
list. I hit a bug that stopped my MDS to start so I took the long way 
with reading the data.


Everything went fine and I can even mount one of my CephFS now. That's 
a big relieve.


But when I start scrub, I just get return code -116 and no scrub is 
initiatet. I didn't find that code in the docs. Can you help me?


[ceph: root@ceph06 /]# ceph tell mds.mds01.ceph06.huavsw scrub start 
recursive
2023-04-29T10:46:36.926+ 7ff676ff5700  0 client.79389355 
ms_handle_reset on v2:192.168.23.66:6800/1133836262
2023-04-29T10:46:36.953+ 7ff676ff5700  0 client.79389361 
ms_handle_reset on v2:192.168.23.66:6800/1133836262

{
    "return_code": -116
}


(I get the same error, no matter what kind of scrub I start)
--
http://www.widhalm.or.at
GnuPG : 6265BAE6 , A84CB603
Threema: H7AV7D33
Telegram, Signal: widha...@widhalm.or.at



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


OpenPGP_signature
Description: OpenPGP digital signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Return code -116 when starting MDS scrub

[Eugen Block]

Hi, according to this thread [1] it means that the scrub path is not  
existing, but you don’t specify a path at all. Can you retry with „/„?  
Do you see anything with damage ls?


[1] https://www.mail-archive.com/ceph-users@lists.ceph.com/msg56062.html

Zitat von Thomas Widhalm :


Hi,

I followed the steps to repair journal and MDS I found here in the  
list. I hit a bug that stopped my MDS to start so I took the long  
way with reading the data.


Everything went fine and I can even mount one of my CephFS now.  
That's a big relieve.


But when I start scrub, I just get return code -116 and no scrub is  
initiatet. I didn't find that code in the docs. Can you help me?


[ceph: root@ceph06 /]# ceph tell mds.mds01.ceph06.huavsw scrub start  
recursive
2023-04-29T10:46:36.926+ 7ff676ff5700  0 client.79389355  
ms_handle_reset on v2:192.168.23.66:6800/1133836262
2023-04-29T10:46:36.953+ 7ff676ff5700  0 client.79389361  
ms_handle_reset on v2:192.168.23.66:6800/1133836262

{
"return_code": -116
}


(I get the same error, no matter what kind of scrub I start)
--
http://www.widhalm.or.at
GnuPG : 6265BAE6 , A84CB603
Threema: H7AV7D33
Telegram, Signal: widha...@widhalm.or.at



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Return code -116 when starting MDS scrub

[Thomas Widhalm]

Hi,

I followed the steps to repair journal and MDS I found here in the list. 
I hit a bug that stopped my MDS to start so I took the long way with 
reading the data.


Everything went fine and I can even mount one of my CephFS now. That's a 
big relieve.


But when I start scrub, I just get return code -116 and no scrub is 
initiatet. I didn't find that code in the docs. Can you help me?


[ceph: root@ceph06 /]# ceph tell mds.mds01.ceph06.huavsw scrub start 
recursive
2023-04-29T10:46:36.926+ 7ff676ff5700  0 client.79389355 
ms_handle_reset on v2:192.168.23.66:6800/1133836262
2023-04-29T10:46:36.953+ 7ff676ff5700  0 client.79389361 
ms_handle_reset on v2:192.168.23.66:6800/1133836262

{
"return_code": -116
}


(I get the same error, no matter what kind of scrub I start)
--
http://www.widhalm.or.at
GnuPG : 6265BAE6 , A84CB603
Threema: H7AV7D33
Telegram, Signal: widha...@widhalm.or.at


OpenPGP_signature
Description: OpenPGP digital signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io