[ceph-users] Re: MDS behind on trimming every 4-5 weeks causing issue for ceph filesystem

[Kotresh Hiremath Ravishankar]

Please share the mds per dump as requested. We need to understand what's
happening before suggesting anything.

Thanks & Regards,
Kotresh H R

On Fri, May 17, 2024 at 5:35 PM Akash Warkhade 
wrote:

> @Kotresh Hiremath Ravishankar 
>
> Can you please help on above
>
>
>
> On Fri, 17 May, 2024, 12:26 pm Akash Warkhade, 
> wrote:
>
>> Hi Kotresh,
>>
>>
>> Thanks for the reply.
>> 1)There are no customer configs defined
>> 2) not enabled subtree pinning
>> 3) there were no warning related to rados
>>
>> So wanted to know In order to fix this should we increase default
>> mds_cache_memory_limit from 4Gb to 6Gb or more?
>> Or is there any other solution for this issue?
>>
>
>> On Fri, 17 May, 2024, 12:11 pm Kotresh Hiremath Ravishankar, <
>> khire...@redhat.com> wrote:
>>
>>> Hi,
>>>
>>> ~6K log segments to be trimmed, that's huge.
>>>
>>> 1. Are there any custom configs configured on this setup ?
>>> 2. Is subtree pinning enabled ?
>>> 3. Are there any warnings w.r.t rados slowness ?
>>> 4. Please share the mds perf dump to check for latencies and other stuff.
>>>$ceph tell mds. perf dump
>>>
>>> Thanks and Regards,
>>> Kotresh H R
>>>
>>> On Fri, May 17, 2024 at 11:01 AM Akash Warkhade 
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> We are using rook-ceph with operator 1.10.8 and ceph 17.2.5.
>>>> we are using ceph filesystem with 4 mds i.e 2 active & 2 standby MDS
>>>> every 3-4 weeks filesystem is having issue i.e in ceph status we can see
>>>> below warnings warnings :
>>>>
>>>> 2 MDS reports slow requests
>>>> 2 MDS Behind on Trimming
>>>> mds.myfs-a(mds.1) : behind on trimming (6378/128) max_segments:128,
>>>> num_segments: 6378
>>>> mds.myfs-c(mds.1):  behind on trimming (6560/128) max_segments:128,
>>>> num_segments: 6560
>>>>
>>>> to fix it, we have to restart all MDS pods one by one.
>>>> this is happening every 4-5 weeks.
>>>>
>>>> We have seen many ceph issues related to it on ceph tracker and many
>>>> people
>>>> are suggesting to increase mds_cache_memory_limit
>>>> currently for our cluster *mds_cache_memory_limit* is set to default 4GB
>>>> *mds_log_max_segments* is set to default 128
>>>> Should we increase *mds_cache_memory_limit* to 8GB from default 4GB or
>>>> is
>>>> there any solution to fix this issue permanently?
>>>> ___
>>>> ceph-users mailing list -- ceph-users@ceph.io
>>>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>>>
>>>>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Write issues on CephFS mounted with root_squash

[Kotresh Hiremath Ravishankar]

On Fri, May 17, 2024 at 11:52 AM Nicola Mori  wrote:

> Thank you Kotresh! My cluster is currently on Reef 18.2.2, which should
> be the current version and which is affected. Will the fix be included
> in the next Reef release?
>

Yes, it's already merged to the reef branch, and should be available in the
next reef release.
Please look at https://tracker.ceph.com/issues/62952


> Cheers,
>
> Nicola
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MDS behind on trimming every 4-5 weeks causing issue for ceph filesystem

[Kotresh Hiremath Ravishankar]

Hi,

~6K log segments to be trimmed, that's huge.

1. Are there any custom configs configured on this setup ?
2. Is subtree pinning enabled ?
3. Are there any warnings w.r.t rados slowness ?
4. Please share the mds perf dump to check for latencies and other stuff.
   $ceph tell mds. perf dump

Thanks and Regards,
Kotresh H R

On Fri, May 17, 2024 at 11:01 AM Akash Warkhade 
wrote:

> Hi,
>
> We are using rook-ceph with operator 1.10.8 and ceph 17.2.5.
> we are using ceph filesystem with 4 mds i.e 2 active & 2 standby MDS
> every 3-4 weeks filesystem is having issue i.e in ceph status we can see
> below warnings warnings :
>
> 2 MDS reports slow requests
> 2 MDS Behind on Trimming
> mds.myfs-a(mds.1) : behind on trimming (6378/128) max_segments:128,
> num_segments: 6378
> mds.myfs-c(mds.1):  behind on trimming (6560/128) max_segments:128,
> num_segments: 6560
>
> to fix it, we have to restart all MDS pods one by one.
> this is happening every 4-5 weeks.
>
> We have seen many ceph issues related to it on ceph tracker and many people
> are suggesting to increase mds_cache_memory_limit
> currently for our cluster *mds_cache_memory_limit* is set to default 4GB
> *mds_log_max_segments* is set to default 128
> Should we increase *mds_cache_memory_limit* to 8GB from default 4GB or is
> there any solution to fix this issue permanently?
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Write issues on CephFS mounted with root_squash

[Kotresh Hiremath Ravishankar]

Hi Nicola,

Yes, this issue is already fixed in main [1] and the quincy backport is
still pending to be merged. Hopefully will be available
in the next Quincy release.

[1] https://github.com/ceph/ceph/pull/48027
[2] https://github.com/ceph/ceph/pull/54469

Thanks and Regards,
Kotresh H R




On Wed, May 15, 2024 at 7:51 PM Fabien Sirjean 
wrote:

> Hi,
>
> We have the same issue. It seems to come from this bug :
> https://access.redhat.com/solutions/6982902
>
> We had to disable root_squash, which of course is a huge issue...
>
> Cheers,
>
> Fabien
>
>
> On 5/15/24 12:54, Nicola Mori wrote:
> > Dear Ceph users,
> >
> > I'm trying to export a CephFS with the root_squash option. This is the
> > client configuration:
> >
> > client.wizardfs_rootsquash
> > key: 
> > caps: [mds] allow rw fsname=wizardfs root_squash
> > caps: [mon] allow r fsname=wizardfs
> > caps: [osd] allow rw tag cephfs data=wizardfs
> >
> > I can mount it flawlessly on several machines using the kernel driver,
> > but when a machine writes on it then the content seems fine from the
> > writing machine but it's not actually written on disk since other
> > machines just see an empty file:
> >
> > [12:43 mori@stryke ~]$ echo test > /wizard/ceph/software/el9/test
> > [12:43 mori@stryke ~]$ ll /wizard/ceph/software/el9/test
> > -rw-r--r-- 1 mori wizard 5 mag 15 12:43 /wizard/ceph/software/el9/test
> > [12:43 mori@stryke ~]$ cat /wizard/ceph/software/el9/test
> > test
> > [12:43 mori@stryke ~]$
> >
> > [mori@fili ~]$ ll /wizard/ceph/software/el9/test
> > -rw-r--r--. 1 mori 1014 0 May 15 06:43 /wizard/ceph/software/el9/test
> > [mori@fili ~]$ cat /wizard/ceph/software/el9/test
> > [mori@fili ~]$
> >
> > Unmounting and then remounting on "stryke" the file is seen as empty,
> > so I guess that the content shown just after the write is only a cache
> > effect and nothing is effectively written on disk. I checked the posix
> > permissions on the folder and I have rw rights from both the machines.
> >
> > All of the above using Ceph 18.2.2 on the cluster (deployed with
> > cephadm) and both the machines. Machine "fili" has kernel 5.14.0 while
> > "stryke" has 6.8.9. The same issue happens consistently also in the
> > reverse direction (writing from "fili" and reading from "stryke"), and
> > also with other machines.
> >
> > Removing the squash_root option the problem vanishes.
> >
> > I don't know what might
> >
> >
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Determine client/inode/dnode source of massive explosion in CephFS metadata pool usage (Red Hat Nautilus CephFS)

[Kotresh Hiremath Ravishankar]

I think you can do the following.

NOTE: If you know the objects that are recently created, you can skip to
step 5

1. List the objects in the metadata pool and copy it to a file
 rados -p  ls > /tmp/metadata_obj_list
2. Prepare a bulk stat script for each object. Unfortunately xargs didn't
work with rados cmd.
 sed "s/^/rados -p  stat /g" /tmp/metadata_obj_list -i
3. bash /tmp/metadata_obj_list
4. Find out the recently created objects based on stat output from step 3
5. Pick any recently created object to map to the directory path
rados -p  getxattr  parent | ceph-dencoder type
'inode_backtrace_t' import - decode dump_json
e.g.,
$rados -p cephfs.a.meta getxattr 10001f9. parent |
ceph-dencoder type 'inode_backtrace_t' import - decode dump_json
{
"ino": 1099511628281,
"ancestors": [
{
"dirino": 1099511628280,
"dname": "dir3",
"version": 4
},
{
"dirino": 1099511627776,
"dname": "dir2",
"version": 13
},
{
"dirino": 1,
"dname": "dir1",
"version": 21
}
],
"pool": 2,
"old_pools": []
}

This is directory object of the path /dir1/dir2/dir3

Thanks and Regards,
Kotresh H R

On Mon, May 13, 2024 at 1:18 PM Eugen Block  wrote:

> I just read your message again, you only mention newly created files,
> not new clients. So my suggestion probably won't help you in this
> case, but it might help others. :-)
>
> Zitat von Eugen Block :
>
> > Hi Paul,
> >
> > I don't really have a good answer to your question, but maybe this
> > approach can help track down the clients.
> >
> > Each MDS client has an average "uptime" metric stored in the MDS:
> >
> > storage01:~ # ceph tell mds.cephfs.storage04.uxkclk session ls
> > ...
> > "id": 409348719,
> > ...
> > "uptime": 844831.115640342,
> > ...
> > "entity_id": "nova-mount",
> > "hostname": "FQDN",
> > "kernel_version": "5.4.0-125-generic",
> > "root": "/openstack-cluster/nova-instances"
> > ...
> >
> > This client has the shortest uptime (9 days), it was a compute node
> > which was integrated into openstack 9 days ago. I don't know your
> > CephFS directory structure, could this help identify the client in
> > your case?
> >
> > Regards,
> > Eugen
> >
> >
> > Zitat von Paul Browne :
> >
> >> Hello Ceph users,
> >>
> >> We've recently seen a very massive uptick in the stored capacity of
> >> our CephFS metadata pool, 150X the raw stored capacity used in a
> >> very short timeframe of only 48 hours or so. The number of stored
> >> objects rose by ~1.5 million or so in that timeframe (attached PNG
> >> shows the increase)
> >>
> >> What I'd really like to be able to determine, but haven't yet
> >> figured out how, is to map these newly stored objects (over this
> >> limited time window) to inodes/dnodes in the filesystem and from
> >> there to individual namespaces being used in the filesystem.
> >>
> >> This should then allow me to track back the increased usage to
> >> specific projects using the filesystem for research data storage
> >> and give them a mild warning about possibly exhausting the
> >> available metadata pool capacity.
> >>
> >> Would anyone know if there's any capability in CephFs to do
> >> something like this, specifically in Nautilus (being run here as
> >> Red Hat Ceph Storage 4)?
> >>
> >> We've scheduled upgrades to later RHCS releases, but I'd like the
> >> cluster and CephFS state to be in a better place first if possible.
> >>
> >> Thanks,
> >> Paul Browne
> >>
> >> [cid:e87ad248-6621-4e9d-948b-da4428f8dbb8]
> >>
> >> ***
> >> Paul Browne
> >> Research Computing Platforms
> >> University Information Services
> >> Roger Needham Building
> >> JJ Thompson Avenue
> >> University of Cambridge
> >> Cambridge
> >> United Kingdom
> >> E-Mail: pf...@cam.ac.uk
> >> Tel: 0044-1223-746548
> >> ***
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: label or pseudo name for cephfs volume path

[Kotresh Hiremath Ravishankar]

On Sat, May 11, 2024 at 6:04 AM Adiga, Anantha 
wrote:

> Hi,
>
> Under the circumstance that a ceph fs subvolume has   to be recreated ,
> the  uuid will change  and   we have to change  all  sources  that
> reference  the volume path.
>
> Is there a way to provide  a  label /tag  to the volume path  that can be
> used for  pv_root_path so that we do not have to change  any references OR
> any suggestion  to address this need ?
>

No, the 'subvolume create' cmd doesn't take the existing volume path.  But
you can create (recreate) it with the same subvolume name.
Could you explain a bit more on the circumstance that requires this ?
Is the expectation to keep the below path unchanged with new subvolume
creation ?
pv_root_path:
/volumes/cephfs_data_pool_ec21_subvolumegroup/cluster_inventory_subvolume/ecfdf977-1fdb-4474-b8dd-c3bedb42620e
You mentioned about uuid changing. The path also contains subvolume name
too. What exact subvolume operations are involved in the use case?



>
> Spec containing  subvolume path:
> k8s_rook_pv:
>   - name: inventory-pv
> subvolumegroup: cephfs_data_pool_ec21_subvolumegroup
> subvolume: cluster_inventory_subvolume
> data_pool: cephfs_data_pool_ec21
> size: 10Gi
> pvc_name: prometheus-inventory-pvc
> pv_root_path:
> /volumes/cephfs_data_pool_ec21_subvolumegroup/cluster_inventory_subvolume/ecfdf977-1fdb-4474-b8dd-c3bedb42620e
> mode: ReadWriteMany
>
>
> Thank you,
> Anantha
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Can setting mds_session_blocklist_on_timeout to false minize the session eviction?

[Kotresh Hiremath Ravishankar]

On Tue, Mar 26, 2024 at 7:30 PM Yongseok Oh 
wrote:

> Hi,
>
> CephFS is provided as a shared file system service in a private cloud
> environment of our company, LINE. The number of sessions is approximately
> more than 5,000, and session evictions occur several times a day. When
> session eviction occurs, the message 'Cannot send after transport endpoint
> shutdown' or 'Permission denied' is displayed and file system access is not
> possible. Our users are very uncomfortable with this issue. In particular,
> there are no special problems such as network connection or CPU usage. When
> I access the machine and take a close look, there are no special problems.
> After this, users feel the inconvenience of having to perform umount/mount
> tasks and run the application again. In a Kubernetes environment, recovery
> is a bit more complicated, which causes a lot of frustration.
>
> I tested that by setting the mds_session_blocklist_on_timeout and
> mds_session_blocklist_on_evict options to false and setting
> client_reconnect_stale to true on the client side, the file system can be
> accessed even if eviction occurs. It seemed like there was no major problem
> accessing the file system as the session was still attached.


> What I'm curious about is if I turn on the above option, will there be any
> other side effects? For example, should I take some action if the integrity
> of the file is broken or if there is an issue on the mds side? I am asking
> a question because there are no details regarding this in the official
> CephFS documentation.
>

The suggestion you are following is documented here
https://docs.ceph.com/en/latest/cephfs/eviction/#advanced-configuring-blocklisting

As you mentioned rightly, it doesn't talk about the side effects if any.
But that said, it's kind of a workaround even if it works. Do we know the
exact reason for session eviction from the logs ?
It could be any of these things mentioned here
https://docs.ceph.com/en/latest/cephfs/eviction/#automatic-client-eviction



> Thank you
>
> Yongseok
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Linux Laptop Losing CephFS mounts on Sleep/Hibernate

[Kotresh Hiremath Ravishankar]

I think the client should reconnect when it's out of sleep. Could you
please share the client logs to check what's happening?



On Tue, Mar 26, 2024 at 4:16 AM  wrote:

> Hi All,
>
> So I've got a Ceph Reef Cluster (latest version) with a CephFS system set
> up with a number of directories on it.
>
> On a Laptop (running Rocky Linux (latest version)) I've used fstab to
> mount a number of those directories - all good, everything works, happy
> happy joy joy! :-)
>
> However, when the laptop goes into sleep or hibernate mode (ie when I
> close the lid) and then bring it back out of sleep/hibernate (ie open the
> lid) the CephFS mounts are "not present". The only way to get them back is
> to run `mount -a` as either root or as sudo. This, as I'm sure you'll
> agree, is less than ideal - especially as this is a pilot project for
> non-admin users (ie they won't have access to the root account or sudo on
> their own (corporate) laptops).
>
> So, my question to the combined wisdom of the Community is what's the best
> way to resolve this issue?
>
> I've looked at autofs, and even tried (half-heartedly - it was late, and I
> wanted to go home  :-) ) to get this running, but I'm note sure if this is
> the best way to resolve things.
>
> All help and advice on this greatly appreciated - thank in advance
>
> Cheers
>
> Dulux-Oz
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Understanding subvolumes

[Kotresh Hiremath Ravishankar]

Comments inline.

On Thu, Feb 1, 2024 at 4:51 AM Matthew Melendy  wrote:

> In our department we're getting starting with Ceph 'reef', using Ceph FUSE
> client for our Ubuntu workstations.
>
> So far so good, except I can't quite figure out one aspect of subvolumes.
>
> When I do the commands:
>
> [root@ceph1 ~]# ceph fs subvolumegroup create cephfs csvg
> [root@ceph1 ~]# ceph fs subvolume create cephfs staff csvg --size
> 2
>
> I get these results:
>
> - A subvolume group csvg is created on volume cephfs
> - A subvolume called staff is created in csvg subvolume (like
> /volumes/csvg/staff ) however there is no size limit set at this folder in
> the Ceph dashboard view
> - A folder with an random UUID name is created inside the subvolume staff
> ( like /volumes/csvg/staff/6a1b3de5-f6ab-4878-aea3-3c3c6f96ffcf ); this
> folder does have a size set on it of 2TB
>
> My questions are:
> - what's the purpose of this UUID, and is it a requirement?
>

The UUID directory is essentially the data directory for the user to store
data.
The subvolume directory is used internally to store metadata related to
subvolume
to support all the subvolume operations.

For more detailed information, please go through the following comment in
the code.
https://github.com/ceph/ceph/blob/main/src/pybind/mgr/volumes/fs/operations/versions/subvolume_v2.py#L19C1-L38C8


- which directory should be mounted for my clients, staff/ or staff/{UUID},
> for the size limit to take effect?
>

 The quota (size passed during subvolume creation/or set after creation) is
enforced on the uuid directory not on subvolume
directory. So it should be staff/{uuid}. The idea is to use the 'subvolume
getpath' command and use the returned path to mount. That
should take care of all the things.

- is there any way to hide or disable this UUID for client mounts? (eg in
> /etc/fstab)
>

I didn't quite get this ?


> [root@ceph1 ~]# ceph fs subvolume getpath cephfs staff csvg
> /volumes/csvg/staff/6a1b3de5-f6ab-4878-aea3-3c3c6f96ffcf
>
> [root@ceph1 ~]# ceph fs subvolume ls cephfs csvg
> [
>  {
>  "name": "staff"
>  }
> ]
>
>
>
> --
> Sincerely,
>
>
> Matthew Melendy
>
> IT Services Specialist
> CS System Services Group
> FEC 3550, University of New Mexico
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: CephFS metadata outgrow DISASTER during recovery

[Kotresh Hiremath Ravishankar]

Hi Jakub,

Comments inline.

On Tue, Jul 25, 2023 at 11:03 PM Jakub Petrzilka 
wrote:

> Hello everyone!
>
> Recently we had a very nasty incident with one of our CEPH storages.
>
> During basic backfill recovery operation due to faulty disk CephFS
> metadata started growing exponentially until they used all available space
> and whole cluster DIED. Usage graph screenshot in attachment.
>

Missed attaching screenshot?
So there were 12 * 240g SSD disks backing the metadata pool, one of these
disks failed?
Could you please share the recovery steps you did after the faulty disk ?


> Everything was very fast and even when the OSDs were marked full they
> tripped failsafe and ate all the free blocks, still trying to allocate
> space and completely died without possibility to even start them again.
>

You mean to say that the size of the mds metadata pool grew exponentially
than the allocated size and mds process eventually died ?


> Only solution was to copy whole bluestore to bigger SSD and resize
> underlying BS device. Just about 1/3 was able to start after moving but it
> was enough since we have very redundant settings for cephfs metadata.
> Basically metadata were moved from 12x 240g SSD to 12x 500GB SSD to have
> enough space to start again.
>
> Brief info about the cluster:
> - CephFS data are stored on ~500x 8TB SAS HDD using 10+2 ECC in 18 hosts.
> - CephFS metadata are stored on ~12x 500GB SAS/SATA SSD using 5x
> replication on 6 hosts.
> - Version was one of the latest 16.x.x Pacific at the time of the incident.
> - 3x Mon+mgr and 2 active and 2 hot standby MDS are on separate virtual
> servers.
> - typical file size to be stored is from hundreds of MBs to tens of GBs.
> - this cluster is not the biggest, not having the most HDDs, no special
> config, I simply see nothing special about this cluster.
>
> During investigation I found out the following:
> - Metadata are outgrowing any time recovery is running on any of
> maintained clusters (~15 clusters of different usages and sizes) but not
> this much, this was an extreme situation.
> - after recovery finish size went fine again.
> - i think there is slight correlation with recovery width (objects to be
> touched by recovery in order to recovery everything) and recovery (time)
> length. But i have no proof.
> - nothing much else
>
> I would like to find out why this happened because i think this can happen
> again sometime and someone might lose some data if they have less luck.
> Any ideas are appreciated, or even info if anyone have seen any similar
> behavior or if i am the only one struggling with issue like this :)
>
> Kind regards,
>
> Jakub Petrzilka
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
Thanks and Regards,
Kotresh H R
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef v18.1.0 QE Validation status

[Kotresh Hiremath Ravishankar]

fs approved.

On Fri, Jun 2, 2023 at 2:54 AM Yuri Weinstein  wrote:

> Still awaiting for approvals:
>
> rados - Radek
> fs - Kotresh and Patrick
>
> upgrade/pacific-x - good as is, Laura?
> upgrade/quicny-x - good as is, Laura?
> upgrade/reef-p2p - N/A
> powercycle - Brad
>
> On Tue, May 30, 2023 at 9:50 AM Yuri Weinstein 
> wrote:
> >
> > Details of this release are summarized here:
> >
> > https://tracker.ceph.com/issues/61515#note-1
> > Release Notes - TBD
> >
> > Seeking approvals/reviews for:
> >
> > rados - Neha, Radek, Travis, Ernesto, Adam King (we still have to
> > merge https://github.com/ceph/ceph/pull/51788 for
> > the core)
> > rgw - Casey
> > fs - Venky
> > orch - Adam King
> > rbd - Ilya
> > krbd - Ilya
> > upgrade/octopus-x - deprecated
> > upgrade/pacific-x - known issues, Ilya, Laura?
> > upgrade/reef-p2p - N/A
> > clients upgrades - not run yet
> > powercycle - Brad
> > ceph-volume - in progress
> >
> > Please reply to this email with approval and/or trackers of known
> > issues/PRs to address them.
> >
> > gibba upgrade was done and will need to be done again this week.
> > LRC upgrade TBD
> >
> > TIA
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Question about xattr and subvolumes

[Kotresh Hiremath Ravishankar]

On Tue, Jun 6, 2023 at 4:30 PM Dario Graña  wrote:

> Hi,
>
> I'm installing a new instance (my first) of Ceph. Our cluster runs
> AlmaLinux9 + Quincy. Now I'm dealing with CephFS and quotas. I read
> documentation about setting up quotas with virtual attributes (xattr) and
> creating volumes and subvolumes with a prefixed size. I cannot distinguish
> which is the best option for us.
>

Creating a volume would create a fs and subvolumes are essentially
directories inside the fs which are managed through
mgr subvolume APIs. The subvolumes are introduced for openstack and
openshift use case which expect these subvolumes
to be programmatically managed via APIs.

Answering the quota question, in cephfs, quota is set using the virtual
xattr. The subvolume creation with size essentially
uses the same virtual xattr interface to set the quota size.


> Currently we create a directory with a project name and some subdirectories
> inside.
>

You can explore subvolumegroup and subvolume mgr APIs if it fits your use
case. Please note that it's mainly designed for
openstack/openshift kind of use cases where each subvolume is per PVC and
the data distinction is maintained e.g., there won't
be hardlinks created across the subvolumes.


> I would like to understand the difference between both options.
>
> Thanks in advance.
>
> --
> Dario Graña
> PIC (Port d'Informació Científica)
> Campus UAB, Edificio D
> E-08193 Bellaterra, Barcelona
> http://www.pic.es
> Avis - Aviso - Legal Notice: http://legal.ifae.es
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MDS recovery

[Kotresh Hiremath Ravishankar]

Hi,

First of all I would suggest upgrading your cluster on one of the supported
releases.

I think full recovery is recommended to get back the mds.

1. Stop the mdses and all the clients.

2. Fail the fs.

a. ceph fs fail 
3. Backup the journal: (If the below command fails, make rados level copy
using http://tracker.ceph.com/issues/9902). Since the mds is corrupted, we
can skip this too ?

# cephfs-journal-tool journal export backup.bin

4. Cleanup up ancillary data generated during if any previous recovery.

# cephfs-data-scan cleanup []

5. Recover_dentries, reset session, and reset_journal:

# cephfs-journal-tool --rank :0 event recover_dentries list

# cephfs-table-tool :all reset session

# cephfs-journal-tool --rank :0 journal reset

6. Execute scan_extents on each of the x4 tools pods in parallel:

# cephfs-data-scan scan_extents --worker_n 0 --worker_m 4 --filesystem
 

# cephfs-data-scan scan_extents --worker_n 1 --worker_m 4 --filesystem
 

# cephfs-data-scan scan_extents --worker_n 2 --worker_m 4 --filesystem
 

# cephfs-data-scan scan_extents --worker_n 3 --worker_m 4 --filesystem
 

 7. Execute scan_inodes on each of the x4 tools pods in parallel:

# cephfs-data-scan scan_inodes --worker_n 0 --worker_m 4 --filesystem
 

# cephfs-data-scan scan_inodes --worker_n 1 --worker_m 4 --filesystem
 

# cephfs-data-scan scan_inodes --worker_n 2 --worker_m 4 --filesystem
 

# cephfs-data-scan scan_inodes --worker_n 3 --worker_m 4 --filesystem
 

 8. scan_links:

# cephfs-data-scan scan_links --filesystem 

9. Mark the filesystem joinable from pod/rook-ceph-tools:

# ceph fs set  joinable true

10. Startup MDSs

11. Scrub online fs

   # ceph tell mds.- scrub start / recursive
repair

12. Check scrub status:

   # ceph tell mds.-{pick-active-mds| a or b} scrub status

For more information please look into
https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/

Thanks,

Kotresh H R

On Wed, Apr 26, 2023 at 3:08 AM  wrote:

> Hi All,
>
> We have a CephFS cluster running Octopus with three control nodes each
> running an MDS, Monitor, and Manager on Ubuntu 20.04. The OS drive on one
> of these nodes failed recently and we had to do a fresh install, but made
> the mistake of installing Ubuntu 22.04 where Octopus is not available. We
> tried to force apt to use the Ubuntu 20.04 repo when installing Ceph so
> that it would install Octopus, but for some reason Quincy was still
> installed. We re-integrated this node and it seemed to work fine for about
> a week until our cluster reported damage to an MDS daemon and placed our
> filesystem into a degraded state.
>
> cluster:
> id: 692905c0-f271-4cd8-9e43-1c32ef8abd13
> health: HEALTH_ERR
> mons are allowing insecure global_id reclaim
> 1 filesystem is degraded
> 1 filesystem is offline
> 1 mds daemon damaged
> noout flag(s) set
> 161 scrub errors
> Possible data damage: 24 pgs inconsistent
> 8 pgs not deep-scrubbed in time
> 4 pgs not scrubbed in time
> 6 daemons have recently crashed
>
>   services:
> mon: 3 daemons, quorum database-0,file-server,webhost (age 12d)
> mgr: database-0(active, since 4w), standbys: webhost, file-server
> mds: cephfs:0/1 3 up:standby, 1 damaged
> osd: 91 osds: 90 up (since 32h), 90 in (since 5M)
>  flags noout
>
>   task status:
>
>   data:
> pools:   7 pools, 633 pgs
> objects: 169.18M objects, 640 TiB
> usage:   883 TiB used, 251 TiB / 1.1 PiB avail
> pgs: 605 active+clean
>  23  active+clean+inconsistent
>  4   active+clean+scrubbing+deep
>  1   active+clean+scrubbing+deep+inconsistent
>
> We are not sure if the Quincy/Octopus version mismatch is the problem, but
> we are in the process of downgrading this node now to ensure all nodes are
> running Octopus. Before doing that, we ran the following commands to try
> and recover:
>
> $ cephfs-journal-tool --rank=cephfs:all journal export backup.bin
>
> $ sudo cephfs-journal-tool --rank=cephfs:all event recover_dentries
> summary:
>
> Events by type:
>   OPEN: 29589
>   PURGED: 1
>   SESSION: 16
>   SESSIONS: 4
>   SUBTREEMAP: 127
>   UPDATE: 70438
> Errors: 0
>
> $ cephfs-journal-tool --rank=cephfs:0 journal reset:
>
> old journal was 170234219175~232148677
> new journal start will be 170469097472 (2729620 bytes past old end)
> writing journal head
> writing EResetJournal entry
> done
>
> $ cephfs-table-tool all reset session
>
> All of our MDS daemons are down and fail to restart with the following
> errors:
>
> -3> 2023-04-20T10:25:15.072-0700 7f0465069700 -1 log_channel(cluster) log
> [ERR] : journal replay alloc 0x153af79 not in free
> [0x153af7d~0x1e8,0x153b35c~0x1f7,0x153b555~0x2,0x153b559~0x2,0x153b55d~0x2,0x153b561~0x2,0x153b565~0x1de,0x153b938~0x1fd

[ceph-users] Re: MDS stuck in "up:replay"

[Kotresh Hiremath Ravishankar]

Hi Thomas,

I have created the tracker https://tracker.ceph.com/issues/58489 to track
this. Please upload the debug mds logs here.

Thanks,
Kotresh H R

On Wed, Jan 18, 2023 at 4:56 PM Kotresh Hiremath Ravishankar <
khire...@redhat.com> wrote:

> Hi Thomas,
>
> This looks like it requires more investigation than I expected. What's the
> current status ?
> Did the crashed mds come back and become active ?
>
> Increase the debug log level to 20 and share the mds logs. I will create a
> tracker and share it here.
> You can upload the mds logs there.
>
> Thanks,
> Kotresh H R
>
> On Tue, Jan 17, 2023 at 5:34 PM Thomas Widhalm 
> wrote:
>
>> Another new thing that just happened:
>>
>> One of the MDS just crashed out of nowhere.
>>
>>
>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.5/rpm/el8/BUILD/ceph-17.2.5/src/mds/journal.cc:
>> In function 'void EMetaBlob::replay(MDSRank*, LogSegment*,
>> MDPeerUpdate*)' thread 7fccc7153700 time 2023-01-17T10:05:15.420191+
>>
>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.5/rpm/el8/BUILD/ceph-17.2.5/src/mds/journal.cc:
>> 1625: FAILED ceph_assert(g_conf()->mds_wipe_sessions)
>>
>>   ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy
>> (stable)
>>   1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x135) [0x7fccd759943f]
>>   2: /usr/lib64/ceph/libceph-common.so.2(+0x269605) [0x7fccd7599605]
>>   3: (EMetaBlob::replay(MDSRank*, LogSegment*, MDPeerUpdate*)+0x5e5c)
>> [0x55fb2b98e89c]
>>   4: (EUpdate::replay(MDSRank*)+0x40) [0x55fb2b98f5a0]
>>   5: (MDLog::_replay_thread()+0x9b3) [0x55fb2b915443]
>>   6: (MDLog::ReplayThread::entry()+0x11) [0x55fb2b5d1e31]
>>   7: /lib64/libpthread.so.0(+0x81ca) [0x7fccd65891ca]
>>   8: clone()
>>
>>
>> and
>>
>>
>>
>> *** Caught signal (Aborted) **
>>   in thread 7fccc7153700 thread_name:md_log_replay
>>
>>   ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy
>> (stable)
>>   1: /lib64/libpthread.so.0(+0x12cf0) [0x7fccd6593cf0]
>>   2: gsignal()
>>   3: abort()
>>   4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x18f) [0x7fccd7599499]
>>   5: /usr/lib64/ceph/libceph-common.so.2(+0x269605) [0x7fccd7599605]
>>   6: (EMetaBlob::replay(MDSRank*, LogSegment*, MDPeerUpdate*)+0x5e5c)
>> [0x55fb2b98e89c]
>>   7: (EUpdate::replay(MDSRank*)+0x40) [0x55fb2b98f5a0]
>>   8: (MDLog::_replay_thread()+0x9b3) [0x55fb2b915443]
>>   9: (MDLog::ReplayThread::entry()+0x11) [0x55fb2b5d1e31]
>>   10: /lib64/libpthread.so.0(+0x81ca) [0x7fccd65891ca]
>>   11: clone()
>>   NOTE: a copy of the executable, or `objdump -rdS ` is
>> needed to interpret this.
>>
>> Is what I found in the logs. Since it's referring to log replaying,
>> could this be related to my issue?
>>
>> On 17.01.23 10:54, Thomas Widhalm wrote:
>> > Hi again,
>> >
>> > Another thing I found: Out of pure desperation, I started MDS on all
>> > nodes. I had them configured in the past so I was hoping, they could
>> > help with bringing in missing data even when they were down for quite a
>> > while now. I didn't see any changes in the logs but the CPU on the hosts
>> > that usually don't run MDS just spiked. So high I had to kill the MDS
>> > again because otherwise they kept killing OSD containers. So I don't
>> > really have any new information, but maybe that could be a hint of some
>> > kind?
>> >
>> > Cheers,
>> > Thomas
>> >
>> > On 17.01.23 10:13, Thomas Widhalm wrote:
>> >> Hi,
>> >>
>> >> Thanks again. :-)
>> >>
>> >> Ok, that seems like an error to me. I never configured an extra rank
>> for
>> >> MDS. Maybe that's where my knowledge failed me but I guess, MDS is
>> >> waiting for something that was never there.
>> >>
>> >> Yes, there are two filesystems. Due to "budget restrictions" (it's my
>> >> personal system at home, I configured a second CephFS with only one
>> >> replica for data that could be easily restored.
>> >>
>> >> Here's what I got when turning up the debug level:
>> >>
>> >> Jan 17 10:08:17 ceph05 ceph-mds[1209]: m

[ceph-users] Re: MDS stuck in "up:replay"

[Kotresh Hiremath Ravishankar]

eph-mds[1209]: mds.0.158167 get_task_status
> >> Jan 17 10:08:25 ceph05 ceph-mds[1209]: mds.0.158167
> >> schedule_update_timer_task
> >> Jan 17 10:08:26 ceph05 ceph-mds[1209]: mds.0.cache Memory usage:  total
> >> 372640, rss 57344, heap 207124, baseline 182548, 0 / 3 inodes have caps,
> >> 0 caps, 0 caps per inode
> >> Jan 17 10:08:26 ceph05 ceph-mds[1209]: mds.0.cache cache not ready for
> >> trimming
> >> Jan 17 10:08:26 ceph05 ceph-mds[1209]: mds.0.cache releasing free memory
> >> Jan 17 10:08:26 ceph05 ceph-mds[1209]: mds.0.cache upkeep thread waiting
> >> interval 1.0s
> >> Jan 17 10:08:27 ceph05 ceph-mds[1209]: mds.0.cache Memory usage:  total
> >> 372640, rss 57272, heap 207124, baseline 182548, 0 / 3 inodes have caps,
> >> 0 caps, 0 caps per inode
> >> Jan 17 10:08:27 ceph05 ceph-mds[1209]: mds.0.cache cache not ready for
> >> trimming
> >> Jan 17 10:08:27 ceph05 ceph-mds[1209]: mds.0.cache upkeep thread waiting
> >> interval 1.0s
> >> Jan 17 10:08:27 ceph05 ceph-mds[1209]: mds.0.158167 get_task_status
> >> Jan 17 10:08:27 ceph05 ceph-mds[1209]: mds.0.158167
> >> schedule_update_timer_task
> >> Jan 17 10:08:28 ceph05 ceph-mds[1209]: mds.0.cache Memory usage:  total
> >> 372640, rss 57040, heap 207124, baseline 182548, 0 / 3 inodes have caps,
> >> 0 caps, 0 caps per inode
> >> Jan 17 10:08:28 ceph05 ceph-mds[1209]: mds.0.cache cache not ready for
> >> trimming
> >> Jan 17 10:08:28 ceph05 ceph-mds[1209]: mds.0.cache upkeep thread waiting
> >> interval 1.0s
> >>
> >>
> >> The only thing that gives me hope here is that the line
> >> mds.beacon.mds01.ceph05.pqxmvt Sending beacon up:replay seq 11109 is
> >> chaning its sequence number.
> >>
> >> Anything else I can provide?
> >>
> >> Cheers,
> >> Thomas
> >>
> >> On 17.01.23 06:27, Kotresh Hiremath Ravishankar wrote:
> >>> Hi Thomas,
> >>>
> >>> Sorry, I misread the mds state to be stuck in 'up:resolve' state. The
> >>> mds is stuck in 'up:replay' which means the MDS taking over a failed
> >>> rank.
> >>> This state represents that the MDS is recovering its journal and other
> >>> metadata.
> >>>
> >>> I notice that there are two filesystems 'cephfs' and 'cephfs_insecure'
> >>> and the active mds for both filesystems are stuck in 'up:replay'. The
> >>> mds
> >>> logs shared are not providing much information to infer anything.
> >>>
> >>> Could you please enable the debug logs and pass on the mds logs ?
> >>>
> >>> Thanks,
> >>> Kotresh H R
> >>>
> >>> On Mon, Jan 16, 2023 at 2:38 PM Thomas Widhalm
> >>> mailto:thomas.widh...@netways.de>> wrote:
> >>>
> >>> Hi Kotresh,
> >>>
> >>> Thanks for your reply!
> >>>
> >>> I only have one rank. Here's the output of all MDS I have:
> >>>
> >>> ###
> >>>
> >>> [ceph: root@ceph06 /]# ceph tell mds.mds01.ceph05.pqxmvt status
> >>> 2023-01-16T08:55:26.055+ 7f3412ffd700  0 client.61249926
> >>> ms_handle_reset on v2:192.168.23.65:6800/2680651694
> >>> <http://192.168.23.65:6800/2680651694>
> >>> 2023-01-16T08:55:26.084+ 7f3412ffd700  0 client.61299199
> >>> ms_handle_reset on v2:192.168.23.65:6800/2680651694
> >>> <http://192.168.23.65:6800/2680651694>
> >>> {
> >>>   "cluster_fsid": "ff6e50de-ed72-11ec-881c-dca6325c2cc4",
> >>>   "whoami": 0,
> >>>   "id": 60984167,
> >>>   "want_state": "up:replay",
> >>>   "state": "up:replay",
> >>>   "fs_name": "cephfs",
> >>>   "replay_status": {
> >>>   "journal_read_pos": 0,
> >>>   "journal_write_pos": 0,
> >>>   "journal_expire_pos": 0,
> >>>   "num_events": 0,
> >>>   "num_segments": 0
> >>>   },
> >>>   "rank_uptime": 150224.982558844,
> >>> 

[ceph-users] Re: MDS stuck in "up:replay"

[Kotresh Hiremath Ravishankar]

> anchor table,9=file layout v2,10=snaprealm v2}
> legacy client fscid: 2
>
> Filesystem 'cephfs' (2)
> fs_name cephfs
> epoch   143850
> flags   12 joinable allow_snaps allow_multimds_snaps
> created 2023-01-14T14:30:05.723421+
> modified2023-01-16T09:00:53.663007+
> tableserver 0
> root0
> session_timeout 60
> session_autoclose   300
> max_file_size   1099511627776
> required_client_features{}
> last_failure0
> last_failure_osd_epoch  12321
> compat  compat={},rocompat={},incompat={1=base v0.20,2=client writeable
> ranges,3=default file layouts on dirs,4=dir inode in separate
> object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds
> uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
> max_mds 1
> in  0
> up  {0=60984167}
> failed
> damaged
> stopped
> data_pools  [4]
> metadata_pool   5
> inline_data disabled
> balancer
> standby_count_wanted1
> [mds.mds01.ceph05.pqxmvt{0:60984167} state up:replay seq 37637 addr
> [v2:192.168.23.65:6800/2680651694,v1:192.168.23.65:6801/2680651694]
> compat {c=[1],r=[1],i=[7ff]}]
>
>
> Filesystem 'cephfs_insecure' (3)
> fs_name cephfs_insecure
> epoch   143849
> flags   12 joinable allow_snaps allow_multimds_snaps
> created 2023-01-14T14:22:46.360062+
> modified2023-01-16T09:00:52.632163+
> tableserver 0
> root0
> session_timeout 60
> session_autoclose   300
> max_file_size   1099511627776
> required_client_features{}
> last_failure0
> last_failure_osd_epoch  12319
> compat  compat={},rocompat={},incompat={1=base v0.20,2=client writeable
> ranges,3=default file layouts on dirs,4=dir inode in separate
> object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds
> uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
> max_mds 1
> in  0
> up  {0=60984134}
> failed
> damaged
> stopped
> data_pools  [7]
> metadata_pool   6
> inline_data disabled
> balancer
> standby_count_wanted1
> [mds.mds01.ceph04.cvdhsx{0:60984134} state up:replay seq 37639 addr
> [v2:192.168.23.64:6800/3930607515,v1:192.168.23.64:6801/3930607515]
> compat {c=[1],r=[1],i=[7ff]}]
>
>
> Standby daemons:
>
> [mds.mds01.ceph07.omdisd{-1:60984161} state up:standby seq 2 addr
> [v2:192.168.23.67:6800/942898192,v1:192.168.23.67:6800/942898192] compat
> {c=[1],r=[1],i=[7ff]}]
> [mds.mds01.ceph06.hsuhqd{-1:60984828} state up:standby seq 1 addr
> [v2:192.168.23.66:6800/4259514518,v1:192.168.23.66:6801/4259514518]
> compat {c=[1],r=[1],i=[7ff]}]
> dumped fsmap epoch 143850
>
> #
>
> [ceph: root@ceph06 /]# ceph fs status
>
> (doesn't come back)
>
> #
>
> All MDS show log lines similar to this one:
>
> Jan 16 10:05:00 ceph04 ceph-mds[1311]: mds.mds01.ceph04.cvdhsx Updating
> MDS map to version 143927 from mon.1
> Jan 16 10:05:05 ceph04 ceph-mds[1311]: mds.mds01.ceph04.cvdhsx Updating
> MDS map to version 143929 from mon.1
> Jan 16 10:05:09 ceph04 ceph-mds[1311]: mds.mds01.ceph04.cvdhsx Updating
> MDS map to version 143930 from mon.1
> Jan 16 10:05:13 ceph04 ceph-mds[1311]: mds.mds01.ceph04.cvdhsx Updating
> MDS map to version 143931 from mon.1
> Jan 16 10:05:20 ceph04 ceph-mds[1311]: mds.mds01.ceph04.cvdhsx Updating
> MDS map to version 143933 from mon.1
> Jan 16 10:05:24 ceph04 ceph-mds[1311]: mds.mds01.ceph04.cvdhsx Updating
> MDS map to version 143935 from mon.1
> Jan 16 10:05:29 ceph04 ceph-mds[1311]: mds.mds01.ceph04.cvdhsx Updating
> MDS map to version 143936 from mon.1
> Jan 16 10:05:33 ceph04 ceph-mds[1311]: mds.mds01.ceph04.cvdhsx Updating
> MDS map to version 143937 from mon.1
> Jan 16 10:05:40 ceph04 ceph-mds[1311]: mds.mds01.ceph04.cvdhsx Updating
> MDS map to version 143939 from mon.1
> Jan 16 10:05:44 ceph04 ceph-mds[1311]: mds.mds01.ceph04.cvdhsx Updating
> MDS map to version 143941 from mon.1
> Jan 16 10:05:49 ceph04 ceph-mds[1311]: mds.mds01.ceph04.cvdhsx Updating
> MDS map to version 143942 from mon.1
>
> Anything else, I can provide?
>
> Cheers and thanks again!
> Thomas
>
> On 16.01.23 06:01, Kotresh Hiremath Ravishankar wrote:
> > Hi Thomas,
> >
> > As the documentation says, the MDS enters up:resolve from |up:replay| if
> > the Ceph file system has multiple ranks (including this one), i.e. it’s
> > not a single active MDS cluster.
> > The MDS is resolving any uncommitted inter-MDS operations. All ranks in
> > the file system must be in this state or later for progress to be made,
> > i.e. no rank can be failed/damaged or |up:replay|.
> >

[ceph-users] Re: MDS stuck in "up:replay"

[Kotresh Hiremath Ravishankar]

Hi Thomas,

As the documentation says, the MDS enters up:resolve from up:replay if the
Ceph file system has multiple ranks (including this one), i.e. it’s not a
single active MDS cluster.
The MDS is resolving any uncommitted inter-MDS operations. All ranks in the
file system must be in this state or later for progress to be made, i.e. no
rank can be failed/damaged or up:replay.

So please check the status of the other active mds if it's failed.

Also please share the mds logs and the output of 'ceph fs dump' and 'ceph
fs status'

Thanks,
Kotresh H R

On Sat, Jan 14, 2023 at 9:07 PM Thomas Widhalm 
wrote:

> Hi,
>
> I'm really lost with my Ceph system. I built a small cluster for home
> usage which has two uses for me: I want to replace an old NAS and I want
> to learn about Ceph so that I have hands-on experience. We're using it
> in our company but I need some real-life experience without risking any
> company or customers data. That's my preferred way of learning.
>
> The cluster consists of 3 Raspberry Pis plus a few VMs running on
> Proxmox. I'm not using Proxmox' built in Ceph because I want to focus on
> Ceph and not just use it as a preconfigured tool.
>
> All hosts are running Fedora (x86_64 and arm64) and during an Upgrade
> from F36 to F37 my cluster suddenly showed all PGs as unavailable. I
> worked nearly a week to get it back online and I learned a lot about
> Ceph management and recovery. The cluster is back but I still can't
> access my data. Maybe you can help me?
>
> Here are my versions:
>
> [ceph: root@ceph04 /]# ceph versions
> {
>  "mon": {
>  "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757)
> quincy (stable)": 3
>  },
>  "mgr": {
>  "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757)
> quincy (stable)": 3
>  },
>  "osd": {
>  "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757)
> quincy (stable)": 5
>  },
>  "mds": {
>  "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757)
> quincy (stable)": 4
>  },
>  "overall": {
>  "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757)
> quincy (stable)": 15
>  }
> }
>
>
> Here's MDS status output of one MDS:
> [ceph: root@ceph04 /]# ceph tell mds.mds01.ceph05.pqxmvt status
> 2023-01-14T15:30:28.607+ 7fb9e17fa700  0 client.60986454
> ms_handle_reset on v2:192.168.23.65:6800/2680651694
> 2023-01-14T15:30:28.640+ 7fb9e17fa700  0 client.60986460
> ms_handle_reset on v2:192.168.23.65:6800/2680651694
> {
>  "cluster_fsid": "ff6e50de-ed72-11ec-881c-dca6325c2cc4",
>  "whoami": 0,
>  "id": 60984167,
>  "want_state": "up:replay",
>  "state": "up:replay",
>  "fs_name": "cephfs",
>  "replay_status": {
>  "journal_read_pos": 0,
>  "journal_write_pos": 0,
>  "journal_expire_pos": 0,
>  "num_events": 0,
>  "num_segments": 0
>  },
>  "rank_uptime": 1127.54018615,
>  "mdsmap_epoch": 98056,
>  "osdmap_epoch": 12362,
>  "osdmap_epoch_barrier": 0,
>  "uptime": 1127.957307273
> }
>
> It's staying like that for days now. If there was a counter moving, I
> just would wait but it doesn't change anything and alle stats says, the
> MDS aren't working at all.
>
> The symptom I have is that Dashboard and all other tools I use say, it's
> more or less ok. (Some old messages about failed daemons and scrubbing
> aside). But I can't mount anything. When I try to start a VM that's on
> RDS I just get a timeout. And when I try to mount a CephFS, mount just
> hangs forever.
>
> Whatever command I give MDS or journal, it just hangs. The only thing I
> could do, was take all CephFS offline, kill the MDS's and do a "ceph fs
> reset  --yes-i-really-mean-it". After that I rebooted all
> nodes, just to be sure but I still have no access to data.
>
> Could you please help me? I'm kinda desperate. If you need any more
> information, just let me know.
>
> Cheers,
> Thomas
>
> --
> Thomas Widhalm
> Lead Systems Engineer
>
> NETWAYS Professional Services GmbH | Deutschherrnstr. 15-19 | D-90429
> Nuernberg
> Tel: +49 911 92885-0 | Fax: +49 911 92885-77
> CEO: Julian Hein, Bernd Erk | AG Nuernberg HRB34510
> https://www.netways.de | thomas.widh...@netways.de
>
> ** stackconf 2023 - September - https://stackconf.eu **
> ** OSMC 2023 - November - https://osmc.de **
> ** New at NWS: Managed Database - https://nws.netways.de/managed-database
> **
> ** NETWAYS Web Services - https://nws.netways.de **
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [ext] Copying large file stuck, two cephfs-2 mounts on two cluster

[Kotresh Hiremath Ravishankar]

Created a tracker to investigate this further.

https://tracker.ceph.com/issues/58376

On Wed, Jan 4, 2023 at 3:18 PM Kotresh Hiremath Ravishankar <
khire...@redhat.com> wrote:

> Hi Mathias,
>
> I am glad that you could find it's a client related issue and figured a
> way around it.
> I too could reproduce the issue locally i.e. when a client which was
> initially copying the snapshot still
> has access to it even when it's got deleted from the other client. I think
> this needs further investigation.
> I will raise a tracker for the same and share it here.
>
> Thanks and Regards,
> Kotresh H R
>
> On Tue, Jan 3, 2023 at 3:23 PM Kuhring, Mathias <
> mathias.kuhr...@bih-charite.de> wrote:
>
>> Trying to exclude clusters and/or clients might have gotten me on the
>> right track. It might have been a client issue or actually a snapshot
>> retention issue. As it turned out when I tried other routes for the data
>> using a different client, the data was not available anymore since the
>> snapshot had been trimmed.
>>
>> We got behind syncing our snapshots a while ago (due to other issues).
>> And now we are somewhere in between our weekly (16 weeks) and daily (30
>> days) snapshots. So, I assume before we catch up with daily (<30), there is
>> a general risk that snapshots disappear while we are syncing them.
>>
>> The funny/weird thing is though (and why I didn't catch up on this), the
>> particular file (and potentially others) of this trimmed snapshot was
>> apparently still available for the client I initially used for the
>> transfer. I'm wondering if the client somehow cached the data until the
>> snapshot got trimmed. And then just re-tried copying the incompletely
>> cached data.
>>
>
>> Continuing with the next available snapshot, mirroring/syncing is now
>> catching up again. I expect it might happen again once we catch up to the
>> 30-days threshold. If the time point of snapshot trimming falls into the
>> syncinc time frame. But then I know to just cancel/skip the current
>> snapshot and continue with the next one. Syncing time is short enough to
>> get me over the hill then before the next trimming.
>>
>> Note to myself: Next time something similar things happens, check if
>> different clients AND different snapshots or original data behave the same.
>>
>> On 12/22/2022 4:27 PM, Kuhring, Mathias wrote:
>>
>> Dear ceph community,
>>
>>
>>
>> We have two ceph cluster of equal size, one main and one mirror, both
>> using cephadm and on version
>>
>> ceph version 17.2.1 (ec95624474b1871a821a912b8c3af68f8f8e7aa1) quincy
>> (stable)
>>
>>
>>
>> We are stuck with copying a large file (~ 64G) between the CephFS file
>> systems of the two clusters.
>>
>>
>> The source path is a snapshot (i.e. something like
>> /my/path/.snap/schedule_some-date/…).
>> But I don't think that should make any difference.
>>
>>
>>
>> First, I was thinking that I need to adapt some rsync parameters to work
>> better with bigger files on CephFS.
>>
>> But when confirming by just copying the file with cp, the transfer get's
>> also stuck.
>>
>> Without any error message, the process just keeps running (rsync or cp).
>>
>> But the file size on the target doesn't increase anymore at some point
>> (almost 85%).
>>
>>
>>
>> Main:
>>
>> -rw--- 1 cockpit-ws printadmin 68360698297 16. Nov 13:40
>> LB22_2764_dragen.bam
>>
>>
>>
>> Mirror:
>>
>> -rw--- 1 root root 58099499008 22. Dez 15:54 LB22_2764_dragen.bam
>>
>>
>>
>> Our CephFS file size limit is with 10 TB more than generous.
>> And as far as I know from clients there are indeed files in TB ranges on
>> the cluster without issues.
>>
>>
>>
>> I don't know if this is the file's fault or if this is some issue with
>> either of the CephFS' or cluster.
>>
>> And I don't know how to look and troubleshoot this.
>>
>> Can anybody give me a tip where I can start looking and debug these kind
>> of issues?
>>
>>
>>
>> Thank you very much.
>>
>>
>>
>> Best Wishes,
>>
>> Mathias
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
>> To unsubscribe send an email to ceph-users-le...@ceph.io> ceph-users-le...@ceph.io>
>>
>>
>> --
>> Mathias Kuhring
>>
>> Dr. rer. nat.
>> Bioinformatician
>> HPC & Core Unit Bioinformatics
>> Berlin Institute of Health at Charité (BIH)
>>
>> E-Mail:  mathias.kuhr...@bih-charite.de> mathias.kuhr...@bih-charite.de>
>> Mobile: +49 172 3475576
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [ext] Copying large file stuck, two cephfs-2 mounts on two cluster

[Kotresh Hiremath Ravishankar]

Hi Mathias,

I am glad that you could find it's a client related issue and figured a way
around it.
I too could reproduce the issue locally i.e. when a client which was
initially copying the snapshot still
has access to it even when it's got deleted from the other client. I think
this needs further investigation.
I will raise a tracker for the same and share it here.

Thanks and Regards,
Kotresh H R

On Tue, Jan 3, 2023 at 3:23 PM Kuhring, Mathias <
mathias.kuhr...@bih-charite.de> wrote:

> Trying to exclude clusters and/or clients might have gotten me on the
> right track. It might have been a client issue or actually a snapshot
> retention issue. As it turned out when I tried other routes for the data
> using a different client, the data was not available anymore since the
> snapshot had been trimmed.
>
> We got behind syncing our snapshots a while ago (due to other issues). And
> now we are somewhere in between our weekly (16 weeks) and daily (30 days)
> snapshots. So, I assume before we catch up with daily (<30), there is a
> general risk that snapshots disappear while we are syncing them.
>
> The funny/weird thing is though (and why I didn't catch up on this), the
> particular file (and potentially others) of this trimmed snapshot was
> apparently still available for the client I initially used for the
> transfer. I'm wondering if the client somehow cached the data until the
> snapshot got trimmed. And then just re-tried copying the incompletely
> cached data.
>

> Continuing with the next available snapshot, mirroring/syncing is now
> catching up again. I expect it might happen again once we catch up to the
> 30-days threshold. If the time point of snapshot trimming falls into the
> syncinc time frame. But then I know to just cancel/skip the current
> snapshot and continue with the next one. Syncing time is short enough to
> get me over the hill then before the next trimming.
>
> Note to myself: Next time something similar things happens, check if
> different clients AND different snapshots or original data behave the same.
>
> On 12/22/2022 4:27 PM, Kuhring, Mathias wrote:
>
> Dear ceph community,
>
>
>
> We have two ceph cluster of equal size, one main and one mirror, both
> using cephadm and on version
>
> ceph version 17.2.1 (ec95624474b1871a821a912b8c3af68f8f8e7aa1) quincy
> (stable)
>
>
>
> We are stuck with copying a large file (~ 64G) between the CephFS file
> systems of the two clusters.
>
>
> The source path is a snapshot (i.e. something like
> /my/path/.snap/schedule_some-date/…).
> But I don't think that should make any difference.
>
>
>
> First, I was thinking that I need to adapt some rsync parameters to work
> better with bigger files on CephFS.
>
> But when confirming by just copying the file with cp, the transfer get's
> also stuck.
>
> Without any error message, the process just keeps running (rsync or cp).
>
> But the file size on the target doesn't increase anymore at some point
> (almost 85%).
>
>
>
> Main:
>
> -rw--- 1 cockpit-ws printadmin 68360698297 16. Nov 13:40
> LB22_2764_dragen.bam
>
>
>
> Mirror:
>
> -rw--- 1 root root 58099499008 22. Dez 15:54 LB22_2764_dragen.bam
>
>
>
> Our CephFS file size limit is with 10 TB more than generous.
> And as far as I know from clients there are indeed files in TB ranges on
> the cluster without issues.
>
>
>
> I don't know if this is the file's fault or if this is some issue with
> either of the CephFS' or cluster.
>
> And I don't know how to look and troubleshoot this.
>
> Can anybody give me a tip where I can start looking and debug these kind
> of issues?
>
>
>
> Thank you very much.
>
>
>
> Best Wishes,
>
> Mathias
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io ceph-users-le...@ceph.io>
>
>
> --
> Mathias Kuhring
>
> Dr. rer. nat.
> Bioinformatician
> HPC & Core Unit Bioinformatics
> Berlin Institute of Health at Charité (BIH)
>
> E-Mail:  mathias.kuhr...@bih-charite.de mathias.kuhr...@bih-charite.de>
> Mobile: +49 172 3475576
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph failing to write data - MDSs read only

[Kotresh Hiremath Ravishankar]

The MDS requests the clients to release caps to trim caches when there is
cache pressure or it
might proactively request the client to release caps in some cases. But the
client is failing to release the
caps soon enough in your case.

Few questions:

1. Have you tuned MDS cache configurations? If so please share.
2. Is this kernel client or fuse client?
3. Could you please share 'session ls' output?
4. Also share the MDS/Client logs.

Sometimes dropping the caches (echo 3 > /proc/sys/vm/drop_caches if it's
kclient) or unmount and mounting
the problematic client  could fix the issue if it's acceptable.

Thanks and Regards,
Kotresh H R

On Thu, Dec 29, 2022 at 4:35 PM Amudhan P  wrote:

> Hi,
>
> Suddenly facing an issue with Ceph cluster I am using ceph version 16.2.6.
> I couldn't find any solution for the issue below.
> Any suggestions?
>
>
> health: HEALTH_WARN
> 1 clients failing to respond to capability release
> 1 clients failing to advance oldest client/flush tid
> 1 MDSs are read only
> 1 MDSs report slow requests
> 1 MDSs behind on trimming
>
>   services:
> mon: 3 daemons, quorum strg-node1,strg-node2,strg-node3 (age 9w)
> mgr: strg-node1.ivkfid(active, since 9w), standbys: strg-node2.unyimy
> mds: 1/1 daemons up, 1 standby
> osd: 32 osds: 32 up (since 9w), 32 in (since 5M)
>
>   data:
> volumes: 1/1 healthy
> pools:   3 pools, 321 pgs
> objects: 13.19M objects, 45 TiB
> usage:   90 TiB used, 85 TiB / 175 TiB avail
> pgs: 319 active+clean
>  2   active+clean+scrubbing+deep
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MDS crashes after evicting client session

[Kotresh Hiremath Ravishankar]

You can find the upstream fix here https://github.com/ceph/ceph/pull/46833

Thanks,
Kotresh HR

On Mon, Sep 26, 2022 at 3:17 PM Dhairya Parmar  wrote:

> Patch for this has already been merged and backported to quincy as well. It
> will be there in the next Quincy release.
>
> On Thu, Sep 22, 2022 at 5:12 PM E Taka <0eta...@gmail.com> wrote:
>
> > Ceph 17.2.3 (dockerized in Ubuntu 20.04)
> >
> > The subject says it. The MDS process always crashes after evicting. ceph
> -w
> > shows:
> >
> > 2022-09-22T13:26:23.305527+0200 mds.ksz-cephfs2.ceph00.kqjdwe [INF]
> > Evicting (and blocklisting) client session 5181680 (
> > 10.149.12.21:0/3369570791)
> > 2022-09-22T13:26:35.729317+0200 mon.ceph00 [INF] daemon
> > mds.ksz-cephfs2.ceph03.vsyrbk restarted
> > 2022-09-22T13:26:36.039678+0200 mon.ceph00 [INF] daemon
> > mds.ksz-cephfs2.ceph01.xybiqv restarted
> > 2022-09-22T13:29:21.000392+0200 mds.ksz-cephfs2.ceph04.ekmqio [INF]
> > Evicting (and blocklisting) client session 5249349 (
> > 10.149.12.22:0/2459302619)
> > 2022-09-22T13:29:32.069656+0200 mon.ceph00 [INF] daemon
> > mds.ksz-cephfs2.ceph01.xybiqv restarted
> > 2022-09-22T13:30:00.000101+0200 mon.ceph00 [INF] overall HEALTH_OK
> > 2022-09-22T13:30:20.710271+0200 mon.ceph00 [WRN] Health check failed: 1
> > daemons have recently crashed (RECENT_CRASH)
> >
> > The crash info of the crashed MDS is:
> > # ceph crash info
> > 2022-09-22T11:26:24.013274Z_b005f3fc-7704-4cfc-96c5-f2a9c993f166
> > {
> >"assert_condition": "!mds->is_any_replay()",
> >"assert_file":
> >
> >
> "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.3/rpm/el8/BUILD/ceph-17.2.3/src/mds/MDLog.cc",
> >
> >"assert_func": "void MDLog::_submit_entry(LogEvent*,
> > MDSLogContextBase*)",
> >"assert_line": 283,
> >"assert_msg":
> >
> >
> "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.3/rpm/el8/BUILD/ceph-17.2.3/src/mds/MDLog.cc:
> > In function 'void MDLog::_submit_entry(LogEvent*, MDSLogContextBase*)'
> > thread 7f76fa8f6700 time
> >
> >
> 2022-09-22T11:26:23.992050+\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.3/rpm/el8/BUILD/ceph-17.2.3/src/mds/MDLog.cc:
> > 283: FAILED ceph_assert(!mds->is_any_replay())\n",
> >"assert_thread_name": "ms_dispatch",
> >"backtrace": [
> >"/lib64/libpthread.so.0(+0x12ce0) [0x7f770231bce0]",
> >"gsignal()",
> >"abort()",
> >"(ceph::__ceph_assert_fail(char const*, char const*, int, char
> > const*)+0x1b0) [0x7f770333bcd2]",
> >"/usr/lib64/ceph/libceph-common.so.2(+0x283e95) [0x7f770333be95]",
> >"(MDLog::_submit_entry(LogEvent*, MDSLogContextBase*)+0x3f)
> > [0x55991905efdf]",
> >"(Server::journal_close_session(Session*, int, Context*)+0x78c)
> > [0x559918d7d63c]",
> >"(Server::kill_session(Session*, Context*)+0x212)
> [0x559918d7dd92]",
> >"(Server::apply_blocklist()+0x10d) [0x559918d7e04d]",
> >"(MDSRank::apply_blocklist(std::set > std::less, std::allocator > const&,
> unsigned
> > int)+0x34) [0x559918d39d74]",
> >"(MDSRankDispatcher::handle_osd_map()+0xf6) [0x559918d3a0b6]",
> >"(MDSDaemon::handle_core_message(boost::intrusive_ptr const>
> > const&)+0x39b) [0x559918d2330b]",
> >"(MDSDaemon::ms_dispatch2(boost::intrusive_ptr
> > const&)+0xc3) [0x559918d23cc3]",
> >"(DispatchQueue::entry()+0x14fa) [0x7f77035c240a]",
> >"(DispatchQueue::DispatchThread::entry()+0x11) [0x7f7703679481]",
> >"/lib64/libpthread.so.0(+0x81ca) [0x7f77023111ca]",
> >"clone()"
> >],
> >"ceph_version": "17.2.3",
> >"crash_id":
> > "2022-09-22T11:26:24.013274Z_b005f3fc-7704-4cfc-96c5-f2a9c993f166",
> >"entity_name": "mds.ksz-cephfs2.ceph03.vsyrbk",
> >"os_id": "centos",
> >"os_name": "CentOS Stream",
> >"os_version": "8",
> >"os_version_id": "8",
> >"process_name": "ceph-mds",
> >"stack_sig":
> > "b75e46941b5f6b7c05a037f9af5d42bb19d82ab7fc6a3c168533fc31a42b4de8",
> >"timestamp": "2022-09-22T11:26:24.013274Z",
> >"utsname_hostname": "ceph03",
> >"utsname_machine": "x86_64",
> >"utsname_release": "5.4.0-125-generic",
> >"utsname_sysname": "Linux",
> >"utsname_version": "#141-Ubuntu SMP Wed Aug 10 13:42:03 UTC 2022"
> > }
> >
> > (Don't be confused by the time information, "ceph -w" is UTC+2, "crash
> > info" is UTC)
> >
> > Should I report this a bug or did I miss something which caused the
> error?
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> >
>
> --
> *Dhairya Parmar*
>
> He/Him/His
>
> Associate Software Engineer, CephFS
>
> Red Hat Inc.

[ceph-users] Re: octopus v15.2.17 QE Validation status

[Kotresh Hiremath Ravishankar]

On Wed, Jul 27, 2022 at 5:02 AM Gregory Farnum  wrote:

> On Tue, Jul 26, 2022 at 3:41 PM Yuri Weinstein 
> wrote:
>
>> Greg, I started testing this PR.
>> What do you want to rerun for it?  Are fs, kcephfs, multimds suites
>> sufficient?
>
>
> We just need to run the mgr/volumes tests — I think those are all in the
> fs suite but Kotresh or Ramana can let us know.
> -Greg
>

Right. Running fs:volumes suite would be sufficient.

>
>
>>
>> On Tue, Jul 26, 2022 at 3:16 PM Gregory Farnum 
>> wrote:
>> >
>> > We can’t do the final release until the recent mgr/volumes security
>> fixes get merged in, though.
>> > https://github.com/ceph/ceph/pull/47236
>> >
>> > On Tue, Jul 26, 2022 at 3:12 PM Ramana Krisna Venkatesh Raja <
>> rr...@redhat.com> wrote:
>> >>
>> >> On Thu, Jul 21, 2022 at 10:28 AM Yuri Weinstein 
>> wrote:
>> >> >
>> >> > Details of this release are summarized here:
>> >> >
>> >> > https://tracker.ceph.com/issues/56484
>> >> > Release Notes - https://github.com/ceph/ceph/pull/47198
>> >> >
>> >> > Seeking approvals for:
>> >> >
>> >> > rados - Neha, Travis, Ernesto, Adam
>> >> > rgw - Casey
>> >> > fs, kcephfs, multimds - Venky, Patrick
>> >>
>> >> fs, kcephfs, multimds approved.
>> >>
>> >> Review of results,
>> >> https://tracker.ceph.com/projects/cephfs/wiki/Octopus#2022-Jul-26
>> >>
>> >> Thanks,
>> >> Ramana
>> >>
>> >> > rbd - Ilya, Deepika
>> >> > krbd  Ilya, Deepika
>> >> > ceph-ansible - Brad pls take a look
>> >> >
>> >> > Please reply to this email with approval and/or trackers of known
>> issues/PRs to address them.
>> >> >
>> >> > PS:  some tests are still running but I will be off-line for several
>> hours and wanted to start the review process.
>> >> >
>> >> > Thx
>> >> > YuriW
>> >> > ___
>> >> > Dev mailing list -- d...@ceph.io
>> >> > To unsubscribe send an email to dev-le...@ceph.io
>> >>
>> >> ___
>> >> Dev mailing list -- d...@ceph.io
>> >> To unsubscribe send an email to dev-le...@ceph.io
>> >>
>>
>>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to capability release

[Kotresh Hiremath Ravishankar]

Is this the kernel client ? If so, could you try dropping the cache as
below ?

# sync; echo 3 > /proc/sys/vm/drop_caches

Mimic is EOL a long time back. Please upgrade to the latest supported version.


Thanks,
Kotresh HR


On Tue, Jul 5, 2022 at 10:21 PM Frank Schilder  wrote:

> Hi all,
>
> I seem to be hit by https://tracker.ceph.com/issues/38491 and have this
> warning now for about 12 hours. I can't find a minimally invasive way to
> fix that in the trackers. Version is mimic.
>
> The client in question is an SMB file server, which is the only server
> accessing the exported directory. I could probably do a "smbcontrol
> close-share ...", but would prefer something less disruptive.
>
> Is there any nice way to make the client and MDS friends again?
>
> Best regards,
> =
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: CephFS Mirroring Extended ACL/Attribute Support

[Kotresh Hiremath Ravishankar]

On Tue, Jul 5, 2022 at 1:06 AM Austin Axworthy 
wrote:

> When syncing using mirroring, the source has extended acls. On the
> destination they are not preserved. Is this intended behavior? Unable to
> find any information on the docs.
>

Yes, this is the intended behavior.

>
>
>
> Is it possible to preserve the acls on the destination?
>

At the moment, cephFS mirroring doesn't support preserving acls.

>
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Active-active MDS networking speed requirements

[Kotresh Hiremath Ravishankar]

On Sat, Apr 9, 2022 at 12:33 AM Vladimir Brik <
vladimir.b...@icecube.wisc.edu> wrote:

> Hello
>
> What speed networking is recommended for active-active MDS
> configurations?
>
I think there is no specific recommendation for active-active MDS. Though
2*1G is sufficient,
it is recommended to use 2*10G for inter MDS communication.
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/5/html/hardware_guide/minimum-hardware-recommendations-for-containerized-ceph_hw


> When I tried active-active configuration on my cluster, the
> MDS servers' bonded 2x1Gbps quickly became saturated.
>
> Is there a way to run two active MDSes on the same server
> (get 2x CPU without saturating the network) and, if just one
> fails, fail-over to two MDSes on another server?
>

Two active MDSes on the same server should work. But if one fails,
failing over onto other two MDSes on the other server is not controllable.


>
> Thanks,
>
> Vlad
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io