[ceph-users] Re: "cephadm version" in reef returns "AttributeError: 'CephadmContext' object has no attribute 'fsid'"

[Eugen Block]

Are the issues you refer to the same as before? I don't think this  
version issue is the root cause, I do see it as well in my test  
cluster(s) but the rest works properly except for the tag issue I  
already reported which you can easily fix by setting the config value  
for the default image  
(https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/LASBJCSPFGDYAWPVE2YLV2ZLF3HC5SLS/#LASBJCSPFGDYAWPVE2YLV2ZLF3HC5SLS). Or are there new issues you  
encountered?


Zitat von Martin Conway :

I just had another look through the issues tracker and found this  
bug already listed.

https://tracker.ceph.com/issues/59428

I need to go back to the other issues I am having and figure out if  
they are related or something different.



Hi

I wrote before about issues I was having with cephadm in 18.2.0  
Sorry, I didn't see the helpful replies because my mail service  
binned the responses.


I still can't get the reef version of cephadm to work properly.

I had updated the system rpm to reef (ceph repo) and also upgraded  
the containerised ceph daemons to reef before my first email.


Both the system package cephadm and the one found at  
/var/lib/ceph/${fsid}/cephadm.* return the same error when running  
"cephadm version"


Traceback (most recent call last):
  File  
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 9468, in  


main()
  File  
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 9456, in  
main

r = ctx.func(ctx)
  File  
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2108, in  
_infer_image

ctx.image = infer_local_ceph_image(ctx, ctx.container_engine.path)
  File  
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2191, in  
infer_local_ceph_image

container_info = get_container_info(ctx, daemon, daemon_name is not None)
  File  
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2154, in  
get_container_info
matching_daemons = [d for d in daemons if daemon_name_or_type(d)  
== daemon_filter and d['fsid'] == ctx.fsid]
  File  
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2154, in  

matching_daemons = [d for d in daemons if daemon_name_or_type(d)  
== daemon_filter and d['fsid'] == ctx.fsid]
  File  
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 217, in  
__getattr__

return super().__getattribute__(name)
AttributeError: 'CephadmContext' object has no attribute 'fsid'

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Problem with upgrade

[Eugen Block]

What do the two Pacific monitors log when you stop the Octopus mon?  
And when all of them are up, which one is the leader?


ceph mon stat --format json | jq


Zitat von Tyler Stachecki :


On Thu, Oct 26, 2023, 8:11 PM Jorge Garcia  wrote:


Oh, I meant that "ceph -s" just hangs. I didn't even try to look at the
I/O. Maybe I can do that, but the "ceph -s" hang just freaked me out.

Also, I know that the recommended order is mon->mgr->osd->mds->rgw, but
when you run mgr on the same hardware as the monitors, it's hard to not
upgrade both at the same time. Particularly if you're upgrading the whole
machine at once. Here's where upgrading to the new container method will
help a lot! FWIW, the managers seem to be running fine.



I recently did something like this, so I understand that it's difficult.
Most of my testing and prep-work was centered around exactly this problem,
which was avoided by first upgrading mons/mgrs to an interim OS while
remaining on Octopus -- solely for the purposes of opening an avenue from
Octopus to Quincy separate from tbe OS upgrade.

In my pre-prod resting, trying to upgrade the mons/mgrs without that middle
step that allowed mgrs to be upgraded separately did result in `ceph -s`
locking up. Client I/O remained non-impacted in this state though.

Maybe look at which mgr is active and/or try stopping all but the Octopus
mgr when stopping the mon as well?

Cheers,
Tyler



On Thu, Oct 26, 2023 at 4:57 PM Tyler Stachecki 
wrote:


On Thu, Oct 26, 2023 at 6:52 PM Jorge Garcia 
wrote:
>
> Hi Tyler,
>
> Maybe you didn't read the full message, but in the message you will
notice that I'm doing exactly that, and the problem just occurred when I
was doing the upgrade from Octopus to Pacific. I'm nowhere near Quincy yet.
The original goal was to move from Nautilus to Quincy, but I have gone to
Octopus (no problems) and now to Pacific (problems).

I did not, apologies -- though do see my second message about ordering
mon/mgr ordering...

When you say "the cluster becomes unresponsive" -- does the client I/O
lock up, or do you mean that `ceph -s` and such hangs?

May help to look to Pacific mons via the asok and see if they respond
in such a state (and their status) if I/O is not locked up and you can
afford to leave it in that state for a couple minutes:
$ ceph daemon mon.name mon_status

Cheers,
Tyler




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: "cephadm version" in reef returns "AttributeError: 'CephadmContext' object has no attribute 'fsid'"

[Martin Conway]

I just had another look through the issues tracker and found this bug already 
listed.
https://tracker.ceph.com/issues/59428

I need to go back to the other issues I am having and figure out if they are 
related or something different.


Hi

I wrote before about issues I was having with cephadm in 18.2.0 Sorry, I didn't 
see the helpful replies because my mail service binned the responses.

I still can't get the reef version of cephadm to work properly.

I had updated the system rpm to reef (ceph repo) and also upgraded the 
containerised ceph daemons to reef before my first email.

Both the system package cephadm and the one found at 
/var/lib/ceph/${fsid}/cephadm.* return the same error when running "cephadm 
version"

Traceback (most recent call last):
  File 
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", 
line 9468, in 
main()
  File 
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", 
line 9456, in main
r = ctx.func(ctx)
  File 
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", 
line 2108, in _infer_image
ctx.image = infer_local_ceph_image(ctx, ctx.container_engine.path)
  File 
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", 
line 2191, in infer_local_ceph_image
container_info = get_container_info(ctx, daemon, daemon_name is not None)
  File 
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", 
line 2154, in get_container_info
matching_daemons = [d for d in daemons if daemon_name_or_type(d) == 
daemon_filter and d['fsid'] == ctx.fsid]
  File 
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", 
line 2154, in 
matching_daemons = [d for d in daemons if daemon_name_or_type(d) == 
daemon_filter and d['fsid'] == ctx.fsid]
  File 
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", 
line 217, in __getattr__
return super().__getattribute__(name)
AttributeError: 'CephadmContext' object has no attribute 'fsid'

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] "cephadm version" in reef returns "AttributeError: 'CephadmContext' object has no attribute 'fsid'"

[Martin Conway]

Hi

I wrote before about issues I was having with cephadm in 18.2.0 Sorry, I didn't 
see the helpful replies because my mail service binned the responses.

I still can't get the reef version of cephadm to work properly.

I had updated the system rpm to reef (ceph repo) and also upgraded the 
containerised ceph daemons to reef before my first email.

Both the system package cephadm and the one found at 
/var/lib/ceph/${fsid}/cephadm.* return the same error when running "cephadm 
version"

Traceback (most recent call last):
  File 
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", 
line 9468, in 
main()
  File 
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", 
line 9456, in main
r = ctx.func(ctx)
  File 
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", 
line 2108, in _infer_image
ctx.image = infer_local_ceph_image(ctx, ctx.container_engine.path)
  File 
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", 
line 2191, in infer_local_ceph_image
container_info = get_container_info(ctx, daemon, daemon_name is not None)
  File 
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", 
line 2154, in get_container_info
matching_daemons = [d for d in daemons if daemon_name_or_type(d) == 
daemon_filter and d['fsid'] == ctx.fsid]
  File 
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", 
line 2154, in 
matching_daemons = [d for d in daemons if daemon_name_or_type(d) == 
daemon_filter and d['fsid'] == ctx.fsid]
  File 
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", 
line 217, in __getattr__
return super().__getattribute__(name)
AttributeError: 'CephadmContext' object has no attribute 'fsid'


I am running into other issues as well, but I think they may point back to this 
issue of "'CephadmContext' object has no attribute 'fsid'"

Any help would be appreciated.

Regards,
Martin Conway
IT and Digital Media Manager
Research School of Physics
Australian National University
Canberra ACT 2601

+61 2 6125 1599
https://physics.anu.edu.au

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Problem with upgrade

[Tyler Stachecki]

On Thu, Oct 26, 2023, 8:11 PM Jorge Garcia  wrote:

> Oh, I meant that "ceph -s" just hangs. I didn't even try to look at the
> I/O. Maybe I can do that, but the "ceph -s" hang just freaked me out.
>
> Also, I know that the recommended order is mon->mgr->osd->mds->rgw, but
> when you run mgr on the same hardware as the monitors, it's hard to not
> upgrade both at the same time. Particularly if you're upgrading the whole
> machine at once. Here's where upgrading to the new container method will
> help a lot! FWIW, the managers seem to be running fine.
>

I recently did something like this, so I understand that it's difficult.
Most of my testing and prep-work was centered around exactly this problem,
which was avoided by first upgrading mons/mgrs to an interim OS while
remaining on Octopus -- solely for the purposes of opening an avenue from
Octopus to Quincy separate from tbe OS upgrade.

In my pre-prod resting, trying to upgrade the mons/mgrs without that middle
step that allowed mgrs to be upgraded separately did result in `ceph -s`
locking up. Client I/O remained non-impacted in this state though.

Maybe look at which mgr is active and/or try stopping all but the Octopus
mgr when stopping the mon as well?

Cheers,
Tyler


> On Thu, Oct 26, 2023 at 4:57 PM Tyler Stachecki 
> wrote:
>
>> On Thu, Oct 26, 2023 at 6:52 PM Jorge Garcia 
>> wrote:
>> >
>> > Hi Tyler,
>> >
>> > Maybe you didn't read the full message, but in the message you will
>> notice that I'm doing exactly that, and the problem just occurred when I
>> was doing the upgrade from Octopus to Pacific. I'm nowhere near Quincy yet.
>> The original goal was to move from Nautilus to Quincy, but I have gone to
>> Octopus (no problems) and now to Pacific (problems).
>>
>> I did not, apologies -- though do see my second message about ordering
>> mon/mgr ordering...
>>
>> When you say "the cluster becomes unresponsive" -- does the client I/O
>> lock up, or do you mean that `ceph -s` and such hangs?
>>
>> May help to look to Pacific mons via the asok and see if they respond
>> in such a state (and their status) if I/O is not locked up and you can
>> afford to leave it in that state for a couple minutes:
>> $ ceph daemon mon.name mon_status
>>
>> Cheers,
>> Tyler
>>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Problem with upgrade

[Jorge Garcia]

Oh, I meant that "ceph -s" just hangs. I didn't even try to look at the
I/O. Maybe I can do that, but the "ceph -s" hang just freaked me out.

Also, I know that the recommended order is mon->mgr->osd->mds->rgw, but
when you run mgr on the same hardware as the monitors, it's hard to not
upgrade both at the same time. Particularly if you're upgrading the whole
machine at once. Here's where upgrading to the new container method will
help a lot! FWIW, the managers seem to be running fine.

I guess, if nothing else works, I can try going:
  Centos7+Octopus to Rocky8+Octopus to Rocky8+Pacific to Rocky9+Pacific to
Rocky9+Quincy
I was just hoping that I could skip the Rocky8 installation altogether...

On Thu, Oct 26, 2023 at 4:57 PM Tyler Stachecki 
wrote:

> On Thu, Oct 26, 2023 at 6:52 PM Jorge Garcia  wrote:
> >
> > Hi Tyler,
> >
> > Maybe you didn't read the full message, but in the message you will
> notice that I'm doing exactly that, and the problem just occurred when I
> was doing the upgrade from Octopus to Pacific. I'm nowhere near Quincy yet.
> The original goal was to move from Nautilus to Quincy, but I have gone to
> Octopus (no problems) and now to Pacific (problems).
>
> I did not, apologies -- though do see my second message about ordering
> mon/mgr ordering...
>
> When you say "the cluster becomes unresponsive" -- does the client I/O
> lock up, or do you mean that `ceph -s` and such hangs?
>
> May help to look to Pacific mons via the asok and see if they respond
> in such a state (and their status) if I/O is not locked up and you can
> afford to leave it in that state for a couple minutes:
> $ ceph daemon mon.name mon_status
>
> Cheers,
> Tyler
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Problem with upgrade

[Tyler Stachecki]

On Thu, Oct 26, 2023 at 6:52 PM Jorge Garcia  wrote:
>
> Hi Tyler,
>
> Maybe you didn't read the full message, but in the message you will notice 
> that I'm doing exactly that, and the problem just occurred when I was doing 
> the upgrade from Octopus to Pacific. I'm nowhere near Quincy yet. The 
> original goal was to move from Nautilus to Quincy, but I have gone to Octopus 
> (no problems) and now to Pacific (problems).

I did not, apologies -- though do see my second message about ordering
mon/mgr ordering...

When you say "the cluster becomes unresponsive" -- does the client I/O
lock up, or do you mean that `ceph -s` and such hangs?

May help to look to Pacific mons via the asok and see if they respond
in such a state (and their status) if I/O is not locked up and you can
afford to leave it in that state for a couple minutes:
$ ceph daemon mon.name mon_status

Cheers,
Tyler
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Problem with upgrade

[Jorge Garcia]

Hi Tyler,

Maybe you didn't read the full message, but in the message you will notice
that I'm doing exactly that, and the problem just occurred when I was doing
the upgrade from Octopus to Pacific. I'm nowhere near Quincy yet. The
original goal was to move from Nautilus to Quincy, but I have gone to
Octopus (no problems) and now to Pacific (problems).

On Thu, Oct 26, 2023 at 3:36 PM Tyler Stachecki 
wrote:

>
> On Thu, Oct 26, 2023, 6:16 PM Jorge Garcia  wrote:
> >
> > from Centos 7 and Nautilus to
> > Rocky 9 and Quincy.
>
> I hate to be the bearer of bad news here, but:
>
> https://docs.ceph.com/en/latest/releases/quincy/#upgrading-from-pre-octopus-releases-like-nautilus
>
> "You *must* first upgrade to Octopus (15.2.z) or Pacific (16.2.z) before
> upgrading to Quincy."
>
> > Is this expected? What am I missing? Thanks for any pointers!
>
> Personally, what I would do in this situation is treat the 2 Quincy mons
> as permanently failed, and recover from the Nautilus one that's still
> working. That is: take a copy of the monmap, then purge the two Quincy mons
> that were prematurely upgraded, and finally rebuild those same 2 mons with
> Centos 7/Nautilus.
>
> Preferably, this is done in some kind of staging environment first for
> validation as you do not want to mess with that last mon at this point.
>
> Cheers,
> Tyler
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Problem with upgrade

[Tyler Stachecki]

On Thu, Oct 26, 2023 at 6:36 PM Tyler Stachecki
 wrote:
>
>
> On Thu, Oct 26, 2023, 6:16 PM Jorge Garcia  wrote:
> >
> > from Centos 7 and Nautilus to
> > Rocky 9 and Quincy.
>
> I hate to be the bearer of bad news here, but:
> https://docs.ceph.com/en/latest/releases/quincy/#upgrading-from-pre-octopus-releases-like-nautilus

Also note that you are supposed to upgrade all of the mons first
(strictly), and *then and only then* the mgrs.

You may also be observing a symptom of this, and the upgraded mons may
be OK. Unsure. Maybe someone else can comment on a Nautilus to Quincy
jump.

> Cheers,
> Tyler
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Problem with upgrade

[Tyler Stachecki]

On Thu, Oct 26, 2023, 6:16 PM Jorge Garcia  wrote:
>
> from Centos 7 and Nautilus to
> Rocky 9 and Quincy.

I hate to be the bearer of bad news here, but:
https://docs.ceph.com/en/latest/releases/quincy/#upgrading-from-pre-octopus-releases-like-nautilus

"You *must* first upgrade to Octopus (15.2.z) or Pacific (16.2.z) before
upgrading to Quincy."

> Is this expected? What am I missing? Thanks for any pointers!

Personally, what I would do in this situation is treat the 2 Quincy mons as
permanently failed, and recover from the Nautilus one that's still working.
That is: take a copy of the monmap, then purge the two Quincy mons that
were prematurely upgraded, and finally rebuild those same 2 mons with
Centos 7/Nautilus.

Preferably, this is done in some kind of staging environment first for
validation as you do not want to mess with that last mon at this point.

Cheers,
Tyler
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Ceph - Error ERANGE: (34) Numerical result out of range

[Pardhiv Karri]

Hi,
Trying to move a node/host under a new SSD root and getting below error.
Has anyone seen it and know the fix? the pg_num and pgp_num are same for
all pools so that is not the issue.

 [root@hbmon1 ~]# ceph osd crush move hbssdhost1 root=ssd
Error ERANGE: (34) Numerical result out of range
 [root@hbmon1 ~]#

Thanks,
Pardhiv
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Problem with upgrade

[Jorge Garcia]

I'm trying to upgrade our 3-monitor cluster from Centos 7 and Nautilus to
Rocky 9 and Quincy. This has been a very slow process of upgrading one
thing, running the cluster for a while, then upgrading the next thing. I
first upgraded to the last Centos 7 and upgraded to Octopus. That worked
fine. Then I was going to upgrade the OS to Rocky 9 while staying on
Octopus, but then found out that Octopus is not available for Rocky 9. So I
broke my own rule and upgraded one of the monitor (and manager) nodes to
Rocky 9 and Pacific, then rejoined it to the cluster. That seemed to work
just fine. Feeling bold, I upgraded the second monitor and manager node to
Rocky 9 and Pacific. That also seemed to work fine, with the cluster
showing all the monitors and managers running. But now, if I shut down the
last "Octopus" monitor, the cluster becomes unresponsive. This only happens
when I shut down the Octopus monitor. If I shut down one of the Pacific
monitors, the cluster keeps responding with the expected:
  "HEALTH_WARN 1/3 mons down"
and then goes back to normal when the monitor process is started again.

Is this expected? What am I missing? Thanks for any pointers!
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Moving devices to a different device class?

[Matt Larson]

Thanks Janne,

 It is good to know that moving the devices over to a new class is a safe
operation.

On Tue, Oct 24, 2023 at 2:16 PM Janne Johansson  wrote:

>
>> The documentation describes that I could set a device class for an OSD
>> with
>> a command like:
>>
>> `ceph osd crush set-device-class CLASS OSD_ID [OSD_ID ..]`
>>
>> Class names can be arbitrary strings like 'big_nvme".  Before setting a
>> new
>> device class to an OSD that already has an assigned device class, should
>> use `ceph osd crush rm-device-class ssd osd.XX`.
>>
>
> Yes, you can re-"name" them by removing old class and setting a new one.
>
>
>> Can I proceed to directly remove these OSDs from the current device class
>> and assign to a new device class? Should they be moved one by one? What is
>> the way to safely protect data from the existing pool that they are mapped
>> to?
>>
>>
> Yes, the PGs on them will be misplaced, so if their pool aims to only use
> "ssd"
> and you re-label them to big-nvme instead, the PGs will look for other
> "ssd"-named
> OSDs to land on, and move themselves if possible. It is a fairly safe
> operation where
> they continue to work, but will try to evacuate the PGs which should not
> be there.
>
> Worst case, your planning is wrong, and the "ssd" OSDs can't accept them,
> and you
> can just undo the relabel and the PGs come back.
>
> --
> May the most significant bit of your life be positive.
>


-- 
Matt Larson, PhD
Madison, WI  53705 U.S.A.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: How to trigger scrubbing in Ceph on-demand ?

[Joe Comeau]

Don't know if this will help you

But we do all our scrubbing manually with cron tasks
always the oldest non-scrubbed pg

And to check on scrubbing we use this - which reports the current
active scrubbing process 
ceph pg ls scrubbing | sort -k18 -k19 | head -n 20
for us a scrub is 5 minutes +/- maybe 3
Deep scrub is 40 minutes +/- 10
all slow hd's


hth  Joe





>>> Jayjeet Chakraborty  10/23/2023 1:59 PM >>>
Hi Reto,

Thanks a lot for the instructions. I tried the same, but still
couldn't
trigger scrubbing deterministically. The first time I initiated
scrubbing,
I saw scrubbing status in ceph -s, but for subsequent times, I didn't
see
any scrubbing status. Do you know what might be going on potentially?
Any
ideas would be appreciated. Thanks.

Best Regards,
*Jayjeet Chakraborty*
Ph.D. Student
Department of Computer Science and Engineering
University of California, Santa Cruz
*Email: jayje...@ucsc.edu *


On Wed, Oct 18, 2023 at 7:47 AM Reto Gysi  wrote:

> Hi
>
> I haven't updated to reef yet. I've tried this on quincy.
>
> # create a testfile on cephfs.rgysi.data pool
> root@zephir:/home/rgysi/misc# echo cephtest123 > cephtest.txt
>
> #list inode of new file
> root@zephir:/home/rgysi/misc# ls -i cephtest.txt
> 1099518867574 cephtest.txt
>
> convert inode value to hex value
> root@zephir:/home/rgysi/misc# printf "%x" 1099518867574
> 16e7876
>
> # search for this value in the rados pool cephfs.rgysi.data, to find
> object(s)
> root@zephir:/home/rgysi/misc# rados -p cephfs.rgysi.data ls | grep
> 16e7876
> 16e7876.
>
> # find pg for the object
> root@zephir:/home/rgysi/misc# ceph osd map cephfs.rgysi.data
> 16e7876.
> osdmap e105365 pool 'cephfs.rgysi.data' (25) object
'16e7876.'
> -> pg 25.ee1befa1 (25.1) -> up ([0,2,8], p0) acting ([0,2,8], p0)
>
> #Initiate a deep-scrub for this pg
> root@zephir:/home/rgysi/misc# ceph pg deep-scrub 25.1
> instructing pg 25.1 on osd.0 to deep-scrub
>
> # check status of scrubbing
> root@zephir:/home/rgysi/misc# ceph pg ls scrubbing
> PGOBJECTS  DEGRADED  MISPLACED  UNFOUND  BYTES   OMAP_BYTES*
>  OMAP_KEYS*  LOG   STATE 
SINCE  VERSION
>REPORTED   UP  ACTING SCRUB_STAMP
> DEEP_SCRUB_STAMP  
 LAST_S
> CRUB_DURATION  SCRUB_SCHEDULING
> 25.137774 00  0  62869823142  
   0
>  0  2402  active+clean+scrubbing+deep  7s 
105365'1178098
>  105365:8066292  [0,2,8]p0  [0,2,8]p0 
2023-10-18T05:17:48.631392+
>  2023-10-08T11:30:58.883164+
>   3  deep scrubbing for 1s
>
>
> Best Regards,
>
> Reto
>
> Am Mi., 18. Okt. 2023 um 16:24 Uhr schrieb Jayjeet Chakraborty <
> jayje...@ucsc.edu>:
>
>> Hi all,
>>
>> Just checking if someone had a chance to go through the scrub
trigger
>> issue
>> above. Thanks.
>>
>> Best Regards,
>> *Jayjeet Chakraborty*
>> Ph.D. Student
>> Department of Computer Science and Engineering
>> University of California, Santa Cruz
>> *Email: jayje...@ucsc.edu *
>>
>>
>> On Mon, Oct 16, 2023 at 9:01 PM Jayjeet Chakraborty

>> wrote:
>>
>> > Hi all,
>> >
>> > I am trying to trigger deep scrubbing in Ceph reef (18.2.0) on
demand
>> on a
>> > set of files that I randomly write to CephFS. I have tried both
invoking
>> > deep-scrub on CephFS using ceph tell and just deep scrubbing a
>> > particular PG. Unfortunately, none of that seems to be working for
me.
>> I am
>> > monitoring the ceph status output, it never shows any scrubbing
>> > information. Can anyone please help me out on this ? In a
nutshell, I
>> need
>> > Ceph to scrub for me anytime I want. I am using Ceph with default
>> configs
>> > for scrubbing. Thanks all.
>> >
>> > Best Regards,
>> > *Jayjeet Chakraborty*
>> > Ph.D. Student
>> > Department of Computer Science and Engineering
>> > University of California, Santa Cruz
>> > *Email: jayjeetc@
ucsc.edu *
>> >
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: quincy v17.2.7 QE Validation status

[Laura Flores]

We have merged a fix for https://tracker.ceph.com/issues/63305, and have
successfully built all distros. Continuing to next steps.

On Wed, Oct 25, 2023 at 12:19 PM Laura Flores  wrote:

> Another outstanding issue is https://tracker.ceph.com/issues/63305, a
> compile-time issue we noticed upon building Debian Bullseye. We have raised
> a small PR to fix the issue, which has been merged and is now undergoing
> testing.
>
> After this, we will be ready to rebuild 17.2.7.
>
> On Wed, Oct 25, 2023 at 11:43 AM Ilya Dryomov  wrote:
>
>> On Mon, Oct 23, 2023 at 5:15 PM Yuri Weinstein 
>> wrote:
>> >
>> > If no one has anything else left, we have all issues resolved and
>> > ready for the 17.2.7 release
>>
>> A last-minute issue with exporter daemon [1][2] necessitated a revert
>> [3].  17.2.7 builds would need to be respinned: since the tag created
>> by Jenkins hasn't been merged and packages haven't been pushed there is
>> no further impact.
>>
>> The lack of test coverage in this area was brought up in the CLT call
>> earlier today.  I have bumped [4] by summarizing the history there.
>>
>> [1] https://github.com/ceph/ceph/pull/54153#discussion_r1369834098
>> [2] https://github.com/ceph/ceph/pull/50749#pullrequestreview-1694336396
>> [3] https://github.com/ceph/ceph/pull/54169
>> [4] https://tracker.ceph.com/issues/59561
>>
>> Thanks,
>>
>> Ilya
>>
>> >
>> > On Mon, Oct 23, 2023 at 8:12 AM Laura Flores 
>> wrote:
>> > >
>> > > Regarding the crash in quincy-p2p (tracked in
>> > > https://tracker.ceph.com/issues/63257), @Prashant Dhange
>> > >  and I evaluated it, and we've concluded it
>> isn't a
>> > > blocker for 17.2.7.
>> > >
>> > > So, quincy-p2p is approved.
>> > >
>> > > Thanks,
>> > > Laura
>> > >
>> > >
>> > >
>> > > On Sat, Oct 21, 2023 at 12:27 AM Venky Shankar 
>> wrote:
>> > >
>> > > > Hi Yuri,
>> > > >
>> > > > On Fri, Oct 20, 2023 at 9:44 AM Venky Shankar 
>> wrote:
>> > > > >
>> > > > > Hi Yuri,
>> > > > >
>> > > > > On Thu, Oct 19, 2023 at 10:48 PM Venky Shankar <
>> vshan...@redhat.com>
>> > > > wrote:
>> > > > > >
>> > > > > > Hi Yuri,
>> > > > > >
>> > > > > > On Thu, Oct 19, 2023 at 9:32 PM Yuri Weinstein <
>> ywein...@redhat.com>
>> > > > wrote:
>> > > > > > >
>> > > > > > > We are still finishing off:
>> > > > > > >
>> > > > > > > - revert PR https://github.com/ceph/ceph/pull/54085, needs
>> smoke
>> > > > suite rerun
>> > > > > > > - removed s3tests https://github.com/ceph/ceph/pull/54078
>> merged
>> > > > > > >
>> > > > > > > Venky, Casey FYI
>> > > > > >
>> > > > > > https://github.com/ceph/ceph/pull/53139 is causing a smoke test
>> > > > > > failure. Details:
>> > > > > > https://github.com/ceph/ceph/pull/53139#issuecomment-1771388202
>> > > > > >
>> > > > > > I've sent a revert for that change -
>> > > > > > https://github.com/ceph/ceph/pull/54108 - will let you know
>> when it's
>> > > > > > ready for testing.
>> > > > >
>> > > > > smoke passes with this revert
>> > > > >
>> > > > >
>> > > >
>> https://pulpito.ceph.com/vshankar-2023-10-19_20:24:36-smoke-wip-vshankar-testing-quincy-20231019.172112-testing-default-smithi/
>> > > > >
>> > > > > fs suite running now...
>> > > >
>> > > > Test results are here -
>> > > >
>> https://tracker.ceph.com/projects/cephfs/wiki/Quincy#2023-October-19
>> > > >
>> > > > Yuri, please merge change - https://github.com/ceph/ceph/pull/54108
>> > > >
>> > > > and consider this as "fs approved".
>> > > >
>> > > > >
>> > > > > >
>> > > > > > >
>> > > > > > > On Wed, Oct 18, 2023 at 9:07 PM Venky Shankar <
>> vshan...@redhat.com>
>> > > > wrote:
>> > > > > > > >
>> > > > > > > > On Tue, Oct 17, 2023 at 12:23 AM Yuri Weinstein <
>> > > > ywein...@redhat.com> wrote:
>> > > > > > > > >
>> > > > > > > > > Details of this release are summarized here:
>> > > > > > > > >
>> > > > > > > > > https://tracker.ceph.com/issues/63219#note-2
>> > > > > > > > > Release Notes - TBD
>> > > > > > > > >
>> > > > > > > > > Issue https://tracker.ceph.com/issues/63192 appears to be
>> > > > failing several runs.
>> > > > > > > > > Should it be fixed for this release?
>> > > > > > > > >
>> > > > > > > > > Seeking approvals/reviews for:
>> > > > > > > > >
>> > > > > > > > > smoke - Laura
>> > > > > > > >
>> > > > > > > > There's one failure in the smoke tests
>> > > > > > > >
>> > > > > > > >
>> > > >
>> https://pulpito.ceph.com/yuriw-2023-10-18_14:58:31-smoke-quincy-release-distro-default-smithi/
>> > > > > > > >
>> > > > > > > > caused by
>> > > > > > > >
>> > > > > > > > https://github.com/ceph/ceph/pull/53647
>> > > > > > > >
>> > > > > > > > (which was marked DNM but got merged). However, it's a test
>> case
>> > > > thing
>> > > > > > > > and we can live with it.
>> > > > > > > >
>> > > > > > > > Yuri mention in slack that he might do another round of
>> > > > build/tests,
>> > > > > > > > so, Yuri, here's the reverted change:
>> > > > > > > >
>> > > > > > > >https://github.com/ceph/ceph/pull/54085
>> > > > > > > >
>> > > > > > > > > rados - 

[ceph-users] Re: owner locked out of bucket via bucket policy

[Wesley Dillingham]

Thank you, this has worked to remove the policy.

Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Wed, Oct 25, 2023 at 5:10 PM Casey Bodley  wrote:

> On Wed, Oct 25, 2023 at 4:59 PM Wesley Dillingham 
> wrote:
> >
> > Thank you, I am not sure (inherited cluster). I presume such an admin
> user created after-the-fact would work?
>
> yes
>
> > Is there a good way to discover an admin user other than iterate over
> all users and retrieve user information? (I presume radosgw-admin user info
> --uid=" would illustrate such administrative access?
>
> not sure there's an easy way to search existing users, but you could
> create a temporary admin user for this repair
>
> >
> > Respectfully,
> >
> > Wes Dillingham
> > w...@wesdillingham.com
> > LinkedIn
> >
> >
> > On Wed, Oct 25, 2023 at 4:41 PM Casey Bodley  wrote:
> >>
> >> if you have an administrative user (created with --admin), you should
> >> be able to use its credentials with awscli to delete or overwrite this
> >> bucket policy
> >>
> >> On Wed, Oct 25, 2023 at 4:11 PM Wesley Dillingham <
> w...@wesdillingham.com> wrote:
> >> >
> >> > I have a bucket which got injected with bucket policy which locks the
> >> > bucket even to the bucket owner. The bucket now cannot be accessed
> (even
> >> > get its info or delete bucket policy does not work) I have looked in
> the
> >> > radosgw-admin command for a way to delete a bucket policy but do not
> see
> >> > anything. I presume I will need to somehow remove the bucket policy
> from
> >> > however it is stored in the bucket metadata / omap etc. If anyone can
> point
> >> > me in the right direction on that I would appreciate it. Thanks
> >> >
> >> > Respectfully,
> >> >
> >> > *Wes Dillingham*
> >> > w...@wesdillingham.com
> >> > LinkedIn 
> >> > ___
> >> > ceph-users mailing list -- ceph-users@ceph.io
> >> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >> >
> >>
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [quincy - 17.2.6] Lua scripting in the rados gateway - HTTP_REMOTE-ADDR missing

[Yuval Lifshitz]

Hi Stephan,
Currently only some of the fields in the HTTP header are exposed, and,
sadly, "REMORE_ADDR" is not.
Created the following PR: https://github.com/ceph/ceph/pull/54211 to expose
all HTTP header fields to lua (note that you can also change them if you
want).

The fix is simple, so I would backport to "quincy" and "reef" once the PR
is merged.

Yuval


On Wed, Oct 25, 2023 at 7:28 PM  wrote:

> Hi Ceph users,
>
> currently I'm using the lua script feature in radosgw to send "put_obj"
> and "get_obj" requests stats to a mongo db.
> So far it's working quite well but I miss a field which is very important
> for us for traffic stats.
> Im looking for the HTTP_REMOTE-ADDR field which is available in the
> ops_log but couldn't find it in here
> https://docs.ceph.com/en/quincy/radosgw/lua-scripting/#request-fields
>
> Does someone know how to get this field via lua script?
>
> Cheers
>
> Stephan
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io