[ceph-users] Re: Monitors not starting, getting "e3 handle_auth_request failed to assign global_id"
We have also had this issue multiple times in 14.2.11 On Tue, Dec 8, 2020, 5:11 PM wrote: > I have same issue. My cluster runing 14.2.11 versions. What is your > version ceph? > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: CentOS
Marc; As if that's not enough confusion (from the FAQ): "Security issues will be updated in CentOS Stream after they are solved in the current RHEL release. Obviously, embargoed security releases can not be publicly released until after the embargo is lifted." Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. dhils...@performair.com www.PerformAir.com -Original Message- From: Marc Roos [mailto:m.r...@f1-outsourcing.eu] Sent: Tuesday, December 8, 2020 3:19 PM To: Dominic Hilsbos; mozes Cc: aKrishna; ceph-users Subject: [ceph-users] Re: CentOS I am confused about that page "Does this mean that CentOS Stream is the RHEL BETA test platform now?" "No, CentOS Stream will be getting fixes and features ahead of RHEL" However this is how wikipedia describes beta: Beta version software is often useful for demonstrations and previews within an organization and to prospective customers. "we expect CentOS Stream to have fewer bugs ... than RHEL until those packages make it into the RHEL release" That looks also contradictory to me. -Original Message- Subject: *SPAM* Re: [ceph-users] Re: CentOS Marc, That video may be out of date. https://centos.org/distro-faq/#q6-will-there-be-separateparallelsimultaneous-streams-for-8-9-10-etc -- Adam On Tue, Dec 8, 2020 at 3:50 PM wrote: > > Marc; > > I'm not happy about this, but RedHat is suggesting that those of us running CentOS for production should move to CentOS Stream. As such, I need to determine if the software I'm running on top of it can be run on Stream. > > Thank you, > > Dominic L. Hilsbos, MBA > Director - Information Technology > Perform Air International Inc. > dhils...@performair.com > www.PerformAir.com > > -Original Message- > From: Marc Roos [mailto:m.r...@f1-outsourcing.eu] > Sent: Tuesday, December 8, 2020 2:02 PM > To: ceph-users; Dominic Hilsbos > Cc: aKrishna > Subject: [ceph-users] Re: CentOS > > > I did not. Thanks for the info. But if I understand this[1] > explanation correctly. CentOS stream is some sort of trial environment > for rhel. So who is ever going to put SDS on such an OS? > > Last post on this blog "But if you read the FAQ, you also learn that > once they start work on RHEL 9, CentOS Stream 8 ceases to exist..." > > [1] > https://www.youtube.com/watch?v=IEEdOogPMY8 > > > > > > > -Original Message- > To: ceph-users@ceph.io > Subject: [ceph-users] CentOS > > All; > > As you may or may not know; this morning RedHat announced the end of > CentOS as a rebuild distribution[1]. "CentOS" will be retired in > favor of the recently announced "CentOS Stream." > > Can Ceph be installed on CentOS Stream? > > Since CentOS Stream is currently at 8, the question really is: Can > Ceph Octopus be installed on CentOS Stream 8? How about Nautilus? > > Thank you, > > Dominic L. Hilsbos, MBA > Director - Information Technology > Perform Air International Inc. > dhils...@performair.com > www.PerformAir.com > > [1: https://blog.centos.org/2020/12/future-is-centos-stream/] > ___ > ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an > email to ceph-users-le...@ceph.io > > ___ > ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an > email to ceph-users-le...@ceph.io > ___ > ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an > email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: CentOS
I am confused about that page "Does this mean that CentOS Stream is the RHEL BETA test platform now?" "No, CentOS Stream will be getting fixes and features ahead of RHEL" However this is how wikipedia describes beta: Beta version software is often useful for demonstrations and previews within an organization and to prospective customers. "we expect CentOS Stream to have fewer bugs ... than RHEL until those packages make it into the RHEL release" That looks also contradictory to me. -Original Message- Subject: *SPAM* Re: [ceph-users] Re: CentOS Marc, That video may be out of date. https://centos.org/distro-faq/#q6-will-there-be-separateparallelsimultaneous-streams-for-8-9-10-etc -- Adam On Tue, Dec 8, 2020 at 3:50 PM wrote: > > Marc; > > I'm not happy about this, but RedHat is suggesting that those of us running CentOS for production should move to CentOS Stream. As such, I need to determine if the software I'm running on top of it can be run on Stream. > > Thank you, > > Dominic L. Hilsbos, MBA > Director - Information Technology > Perform Air International Inc. > dhils...@performair.com > www.PerformAir.com > > -Original Message- > From: Marc Roos [mailto:m.r...@f1-outsourcing.eu] > Sent: Tuesday, December 8, 2020 2:02 PM > To: ceph-users; Dominic Hilsbos > Cc: aKrishna > Subject: [ceph-users] Re: CentOS > > > I did not. Thanks for the info. But if I understand this[1] > explanation correctly. CentOS stream is some sort of trial environment > for rhel. So who is ever going to put SDS on such an OS? > > Last post on this blog "But if you read the FAQ, you also learn that > once they start work on RHEL 9, CentOS Stream 8 ceases to exist..." > > [1] > https://www.youtube.com/watch?v=IEEdOogPMY8 > > > > > > > -Original Message- > To: ceph-users@ceph.io > Subject: [ceph-users] CentOS > > All; > > As you may or may not know; this morning RedHat announced the end of > CentOS as a rebuild distribution[1]. "CentOS" will be retired in > favor of the recently announced "CentOS Stream." > > Can Ceph be installed on CentOS Stream? > > Since CentOS Stream is currently at 8, the question really is: Can > Ceph Octopus be installed on CentOS Stream 8? How about Nautilus? > > Thank you, > > Dominic L. Hilsbos, MBA > Director - Information Technology > Perform Air International Inc. > dhils...@performair.com > www.PerformAir.com > > [1: https://blog.centos.org/2020/12/future-is-centos-stream/] > ___ > ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an > email to ceph-users-le...@ceph.io > > ___ > ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an > email to ceph-users-le...@ceph.io > ___ > ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an > email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Monitors not starting, getting "e3 handle_auth_request failed to assign global_id"
Sorry my cluster is 14.2.11. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Monitors not starting, getting "e3 handle_auth_request failed to assign global_id"
I have same issue. My cluster runing 14.2.11 versions. What is your version ceph? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Upgrade to 15.2.7 fails on mixed x86_64/arm64 cluster
I think you should open an issue on the ceph tracker as it seems the cephadm upgrade workflow doesn't support multi arch container images. docker.io/ceph/ceph:v15.2.7 is a manifest list [1], which depending on the host architecture (x86_64 or ARMv8), will provide you the right container image. docker.io/ceph/ceph manifest references docker.io/ceph/ceph-amd64 and docker.io/ceph/ceph-arm64 container images. So it's expected to have the container image ID 2bc420ddb175 on your x86_64 host and 9a0677fecc08 on ARMv8 host but cephadm doesn't take care of this configuration as the container image ID is compared between two hosts with a different arch [2]. [1] https://hub.docker.com/r/ceph/ceph/tags?page=1=last_updated=v15.2.7 [2] https://github.com/ceph/ceph/blob/master/src/pybind/mgr/cephadm/upgrade.py#L314-L319 ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Larger number of OSDs, cheroot, cherrypy, limits + containers == broken
As far as I know, the issue isn't specific to using container as deployment using packages (rpm or deb) are also affected by the issue (at least CentOS 8 and Ubuntu 20.04 focal) ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: CentOS
Marc, That video may be out of date. https://centos.org/distro-faq/#q6-will-there-be-separateparallelsimultaneous-streams-for-8-9-10-etc -- Adam On Tue, Dec 8, 2020 at 3:50 PM wrote: > > Marc; > > I'm not happy about this, but RedHat is suggesting that those of us running > CentOS for production should move to CentOS Stream. As such, I need to > determine if the software I'm running on top of it can be run on Stream. > > Thank you, > > Dominic L. Hilsbos, MBA > Director - Information Technology > Perform Air International Inc. > dhils...@performair.com > www.PerformAir.com > > -Original Message- > From: Marc Roos [mailto:m.r...@f1-outsourcing.eu] > Sent: Tuesday, December 8, 2020 2:02 PM > To: ceph-users; Dominic Hilsbos > Cc: aKrishna > Subject: [ceph-users] Re: CentOS > > > I did not. Thanks for the info. But if I understand this[1] explanation > correctly. CentOS stream is some sort of trial environment for rhel. So > who is ever going to put SDS on such an OS? > > Last post on this blog "But if you read the FAQ, you also learn that > once they start work on RHEL 9, CentOS Stream 8 ceases to exist..." > > [1] > https://www.youtube.com/watch?v=IEEdOogPMY8 > > > > > > > -Original Message- > To: ceph-users@ceph.io > Subject: [ceph-users] CentOS > > All; > > As you may or may not know; this morning RedHat announced the end of > CentOS as a rebuild distribution[1]. "CentOS" will be retired in favor > of the recently announced "CentOS Stream." > > Can Ceph be installed on CentOS Stream? > > Since CentOS Stream is currently at 8, the question really is: Can Ceph > Octopus be installed on CentOS Stream 8? How about Nautilus? > > Thank you, > > Dominic L. Hilsbos, MBA > Director - Information Technology > Perform Air International Inc. > dhils...@performair.com > www.PerformAir.com > > [1: https://blog.centos.org/2020/12/future-is-centos-stream/] > ___ > ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an > email to ceph-users-le...@ceph.io > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: CentOS
Marc; I'm not happy about this, but RedHat is suggesting that those of us running CentOS for production should move to CentOS Stream. As such, I need to determine if the software I'm running on top of it can be run on Stream. Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. dhils...@performair.com www.PerformAir.com -Original Message- From: Marc Roos [mailto:m.r...@f1-outsourcing.eu] Sent: Tuesday, December 8, 2020 2:02 PM To: ceph-users; Dominic Hilsbos Cc: aKrishna Subject: [ceph-users] Re: CentOS I did not. Thanks for the info. But if I understand this[1] explanation correctly. CentOS stream is some sort of trial environment for rhel. So who is ever going to put SDS on such an OS? Last post on this blog "But if you read the FAQ, you also learn that once they start work on RHEL 9, CentOS Stream 8 ceases to exist..." [1] https://www.youtube.com/watch?v=IEEdOogPMY8 -Original Message- To: ceph-users@ceph.io Subject: [ceph-users] CentOS All; As you may or may not know; this morning RedHat announced the end of CentOS as a rebuild distribution[1]. "CentOS" will be retired in favor of the recently announced "CentOS Stream." Can Ceph be installed on CentOS Stream? Since CentOS Stream is currently at 8, the question really is: Can Ceph Octopus be installed on CentOS Stream 8? How about Nautilus? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. dhils...@performair.com www.PerformAir.com [1: https://blog.centos.org/2020/12/future-is-centos-stream/] ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: How to copy an OSD from one failing disk to another one
For Ceph,this is fortunately not a major issue. Drives failing is considered entirely normal, and Ceph will automatically rebuild your data from redundancy onto a new replacement drive.If You're able to predict the imminent failure of a drive, adding a new drive /OSD will automatically start flowing data onto that drive immediately, thus reducing the time period with decreased redundancy.If You're running with very tight levels of redundancy, You're better off, creating a new OSD on a replacement drive before destroying the old OSD on the failed drive. But if You're running with anything near the recommended/default levels of redundancy, it doesn't really matter in which order you do it. Best regards, Simon Kepp, Kepp Technologies. On Tue, Dec 8, 2020 at 8:59 PM Konstantin Shalygin wrote: > Destroy this OSD, replace disk, deploy OSD. > > > k > > Sent from my iPhone > > > On 8 Dec 2020, at 15:13, huxia...@horebdata.cn wrote: > > > > Hi, dear cephers, > > > > On one ceph i have a failing disk, whose SMART information signals an > impending failure but still availble for reads and writes. I am setting up > a new disk on the same node to replace it. > > What is the best procedure to migrate data (or COPY ) from the failing > OSD to the new one? > > > > Is there any stardard method to copy the OSD from one to another? > > > > best regards, > > > > samuel > > > > > > > > huxia...@horebdata.cn > > ___ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: CentOS
I did not. Thanks for the info. But if I understand this[1] explanation correctly. CentOS stream is some sort of trial environment for rhel. So who is ever going to put SDS on such an OS? Last post on this blog "But if you read the FAQ, you also learn that once they start work on RHEL 9, CentOS Stream 8 ceases to exist..." [1] https://www.youtube.com/watch?v=IEEdOogPMY8 -Original Message- To: ceph-users@ceph.io Subject: [ceph-users] CentOS All; As you may or may not know; this morning RedHat announced the end of CentOS as a rebuild distribution[1]. "CentOS" will be retired in favor of the recently announced "CentOS Stream." Can Ceph be installed on CentOS Stream? Since CentOS Stream is currently at 8, the question really is: Can Ceph Octopus be installed on CentOS Stream 8? How about Nautilus? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. dhils...@performair.com www.PerformAir.com [1: https://blog.centos.org/2020/12/future-is-centos-stream/] ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] CentOS
All; As you may or may not know; this morning RedHat announced the end of CentOS as a rebuild distribution[1]. "CentOS" will be retired in favor of the recently announced "CentOS Stream." Can Ceph be installed on CentOS Stream? Since CentOS Stream is currently at 8, the question really is: Can Ceph Octopus be installed on CentOS Stream 8? How about Nautilus? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. dhils...@performair.com www.PerformAir.com [1: https://blog.centos.org/2020/12/future-is-centos-stream/] ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: How to copy an OSD from one failing disk to another one
Destroy this OSD, replace disk, deploy OSD. k Sent from my iPhone > On 8 Dec 2020, at 15:13, huxia...@horebdata.cn wrote: > > Hi, dear cephers, > > On one ceph i have a failing disk, whose SMART information signals an > impending failure but still availble for reads and writes. I am setting up a > new disk on the same node to replace it. > What is the best procedure to migrate data (or COPY ) from the failing OSD to > the new one? > > Is there any stardard method to copy the OSD from one to another? > > best regards, > > samuel > > > > huxia...@horebdata.cn > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Running Mons on msgrv2/3300 only.
We rebuilt all of our mons in one cluster such that they bind only to port 3300 with msgrv2. Previous to this we were binding to both 6789 and 3300. All of our server and client components are sufficiently new (14.2.x) and we haven’t observed any disruption but I am inquiring if this may be problematic for any unforeseen reason. We don’t intend to have any older clients connecting. https://docs.ceph.com/en/latest/rados/configuration/msgr2/ doesn’t mention much about running with only v2 so I just want to make sure we aren’t setting ourselves up for trouble. Thanks. -- Respectfully, Wes Dillingham Site Reliability Engineer IV Storage Engineering / Ceph wdillingham(at)godaddy.com ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Announcing go-ceph v0.7.0
I'm happy to announce another release of the go-ceph API bindings. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.7.0 Changes in the release are detailed in the link above. The bindings aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. These API bindings require the use of cgo. There are already a few consumers of this library in the wild, including the ceph-csi project. Specific questions, comments, bugs etc are best directed at our github issues tracker. -- John Mulligan phlogistonj...@asynchrono.us jmulli...@redhat.com ___ Dev mailing list -- d...@ceph.io To unsubscribe send an email to dev-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Larger number of OSDs, cheroot, cherrypy, limits + containers == broken
Hi Ken, Thank you for the update! As per: https://github.com/ceph/ceph-container/issues/1748 We implemented the (dropping ulimit to 1024:4096 for mgr) suggested change last night, and on our test cluster of 504 OSDs, being polled by the internal prometheus and our external instance, the mgrs stopped responding and dropped out of the cluster entirely. This is impacting not just metrics, but the mgr itself. I think this is a high priority issue, as metrics are critical for prod, but mgr itself seems to be impacted on a moderately sized cluster. Respectfully, David Orman On Mon, Dec 7, 2020 at 1:50 PM Ken Dreyer wrote: > Thanks for bringing this up. > > We need to update Cheroot in Fedora and EPEL 8. I've opened > https://src.fedoraproject.org/rpms/python-cheroot/pull-request/3 to > get this into Fedora first. > > I've published an el8 RPM at > https://fedorapeople.org/~ktdreyer/bz1868629/ for early testing. I can > bring up a "hello world" cherrypy app with this, but I've not tested > it with Ceph. > > - Ken > > On Mon, Dec 7, 2020 at 9:57 AM David Orman wrote: > > > > Hi, > > > > We have a ceph 15.2.7 deployment using cephadm under podman w/ systemd. > > We've run into what we believe is: > > > > https://github.com/ceph/ceph-container/issues/1748 > > https://tracker.ceph.com/issues/47875 > > > > In our case, eventually the mgr container stops emitting output/logging. > We > > are polling with external prometheus clusters, which is likely what > > triggers the issue, as it appears some amount of time after the container > > is spawned. > > > > Unfortunately, setting limits in the systemd service file for the mgr > > service on the host OS doesn't work, nor does modifying the unit.run file > > which is used to start the container under podman to include the --ulimit > > settings as suggested. Looking inside the container: > > > > lib/systemd/system/ceph-mgr@.service:LimitNOFILE=1048576 > > > > This prevents us from deploying medium to large ceph clusters, so I would > > argue it's a high priority bug that should not be closed, unless there > is a > > workaround that works until EPEL 8 contains the fixed version of cheroot > > and the ceph containers include it. > > > > My understanding is this was fixed in cheroot 8.4.0: > > > > https://github.com/cherrypy/cheroot/issues/249 > > https://github.com/cherrypy/cheroot/pull/301 > > > > Thank you in advance for any suggestions, > > David > > ___ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Ceph on vector machines
Hi all, Just out of curiosity.Considering vector machines are being used in HPC applications to accelerate certain kernels, do you think there are some workloads in Ceph that could be good candidates to be offloaded and accelerated on vector machines ? Thanks in advance. BR ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: rgw index shard much larger than others
Hi Eric & Matt,
I'm working on this again, and was able to reproduce with a versioned
test bucket in v14.2.11. I put a test file "passwd", then deleted it,
then let the lc trim the versions. The exact lc and resulting bi list
are at: https://stikked.web.cern.ch/stikked/view/raw/cc748686
> an automated clean-up is non-trivial but feasible; it would have to take into
> account that an object with the same name as the previously deleted one was
> re-created in the versioned bucket
I've tried various things to remove this nameless entry, but didn't succeed.
(tried adding / removing the same named object, with and without
versioning, with and without the lc enabled. in all cases the unnamed
entry remains.)
Do you have any suggestions on how to remove that entry? Maybe I need
to remove the omap key directly (which will be an interesting
challenge, given that the key starts with 0x80).
Also, it occurred to me that even if I'm able to clean up these
entries, and even with the fix for
https://tracker.ceph.com/issues/46456, we'll still have the problem
that "when the final instance of an object in a versioned bucket is
deleted, but for reasons we do not yet understand, the object was not
fully deleted from the bucket index". So we'll accumulate these zombie
entries, even though they'll now be reshardable.
In other words, I was thinking of resolving our issue by simply
rcloning from our affected bucket to a new bucket, but the 2nd bug
would leave us with a large number of useless index entries.
Should we open a tracker either of these things? (removing unnamed
entries, removing the last index entry of an object)
Best Regards,
Dan
On Fri, Oct 2, 2020 at 10:02 AM Dan van der Ster wrote:
>
> Hi Eric,
>
> So yes we're hit by this. We have around 1.6M entries in shard 0 with
> an empty key, e.g.:
>
> {
> "type": "olh",
> "idx":
> "<80>1001_02/5f/025f8e0fc8234530d6ae7302adf682509f0f7fb68666391122e16d00bd7107e3/2018_11_14/2625203/3034777/metadata.gz",
> "entry": {
> "key": {
> "name": "",
> "instance": ""
> },
> "delete_marker": "false",
> "epoch": 11,
> "pending_log": [],
> "tag": "uhzz6da13ovbr69hhlttdjqmwic4f2v8",
> "exists": "false",
> "pending_removal": "true"
> }
> },
>
> exists is false and pending_removal is true for all of them.
>
> Cheers, Dan
>
> On Thu, Oct 1, 2020 at 11:32 PM Eric Ivancich wrote:
> >
> > Hi Dan,
> >
> > One way to tell would be to do a:
> >
> > radosgw-admin bi list —bucket=
> >
> > And see if any of the lines output contains (perhaps using `grep`):
> >
> > "type": "olh",
> >
> > That would tell you if there were any versioned objects in the bucket.
> >
> > The “fix” we currently have only prevents this from happening in the
> > future. We currently do not have a “fix” that cleans up the bucket index.
> > Like I mentioned — an automated clean-up is non-trivial but feasible; it
> > would have to take into account that an object with the same name as the
> > previously deleted one was re-created in the versioned bucket.
> >
> > I hope that’s informative, if not what you were hoping to hear.
> >
> > Eric
> > --
> > J. Eric Ivancich
> >
> > he / him / his
> > Red Hat Storage
> > Ann Arbor, Michigan, USA
> >
> > On Oct 1, 2020, at 10:53 AM, Dan van der Ster wrote:
> >
> > Thanks Matt and Eric,
> >
> > Sorry for the basic question, but how can I as a ceph operator tell if
> > a bucket is versioned?
> >
> > And for fixing this current situation, I would wait for the fix then
> > reshard?
> > (We want to reshard this bucket anyway because listing perf is way too
> > slow for the user with 512 shards).
> >
> > -- Dan
> >
> >
> > On Thu, Oct 1, 2020 at 4:36 PM Eric Ivancich wrote:
> >
> >
> > Hi Matt and Dan,
> >
> > I too suspect it’s the issue Matt linked to. That bug only affects
> > versioned buckets, so I’m guessing your bucket is versioned, Dan.
> >
> > This bug is triggered when the final instance of an object in a versioned
> > bucket is deleted, but for reasons we do not yet understand, the object was
> > not fully deleted from the bucket index. And then a reshard moves part of
> > the object index to shard 0.
> >
> > Upgrading to a version that included Casey’s fix would mean this situation
> > is not re-created in the future.
> >
> > An automated clean-up is non-trivial but feasible. It would have to take
> > into account that an object with the same name as the previously deleted
> > one was re-created in the versioned bucket.
> >
> > Eric
> >
> > On Oct 1, 2020, at 8:46 AM, Matt Benjamin wrote:
> >
> > Hi Dan,
> >
> > Possibly you're reproducing https://tracker.ceph.com/issues/46456.
> >
> > That explains how the underlying issue worked, I don't remember how a
> > bucked exhibiting this is repaired.
> >
> > Eric?
> >
> > Matt
> >
> >
> > On Thu, Oct 1, 2020 at 8:41 AM Dan van der Ster
[ceph-users] Re: How to copy an OSD from one failing disk to another one
Thanks a lot. I got it. huxia...@horebdata.cn From: Janne Johansson Date: 2020-12-08 13:38 To: huxia...@horebdata.cn CC: ceph-users Subject: Re: [ceph-users] How to copy an OSD from one failing disk to another one "ceph osd set norebalance" "ceph osd set nobackfill" Add new OSD, set osd weight to 0 on old OSD unset the norebalance and nobackfill options, and the cluster will do it all for you. Den tis 8 dec. 2020 kl 13:13 skrev huxia...@horebdata.cn : Hi, dear cephers, On one ceph i have a failing disk, whose SMART information signals an impending failure but still availble for reads and writes. I am setting up a new disk on the same node to replace it. What is the best procedure to migrate data (or COPY ) from the failing OSD to the new one? Is there any stardard method to copy the OSD from one to another? best regards, samuel huxia...@horebdata.cn ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io -- May the most significant bit of your life be positive. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: How to copy an OSD from one failing disk to another one
"ceph osd set norebalance" "ceph osd set nobackfill" Add new OSD, set osd weight to 0 on old OSD unset the norebalance and nobackfill options, and the cluster will do it all for you. Den tis 8 dec. 2020 kl 13:13 skrev huxia...@horebdata.cn < huxia...@horebdata.cn>: > Hi, dear cephers, > > On one ceph i have a failing disk, whose SMART information signals an > impending failure but still availble for reads and writes. I am setting up > a new disk on the same node to replace it. > What is the best procedure to migrate data (or COPY ) from the failing OSD > to the new one? > > Is there any stardard method to copy the OSD from one to another? > > best regards, > > samuel > > > > huxia...@horebdata.cn > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > -- May the most significant bit of your life be positive. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] CfP Software Defined Storage devroom
FOSDEM is a free software event that offers open source communities a place to
meet, share ideas and collaborate. It is well known for being highly
developer-oriented and in the past brought together 8000+ participants from all
over the world. It's home is in the city of Brussels (Belgium).
FOSDEM 2021 will take place as an online event during the weekend of February
6./7. 2021. More details about the event can be found at http://fosdem.org/
** Call For Participation
The Software Defined Storage devroom will go into it's fifth round for talks
around Open Source Software Defined Storage projects, management tools
and real world deployments.
Presentation topics could include but are not limited too:
- Your work on a SDS project like Ceph, Gluster, OpenEBS, CORTX or Longhorn
- Your work on or with SDS related projects like OpenStack SWIFT or Container
Storage Interface
- Management tools for SDS deployments
- Monitoring tools for SDS clusters
** Important dates:
- Dec 27th 2020: submission deadline for talk proposals
- Dec 31st 2020: announcement of the final schedule
- Feb 6th 2021: Software Defined Storage dev room
Talk proposals will be reviewed by a steering committee:
- Niels de Vos (OpenShift Container Storage Developer - Red Hat)
- Jan Fajerski (Ceph Developer - SUSE)
- TBD
Use the FOSDEM 'pentabarf' tool to submit your proposal:
https://penta.fosdem.org/submission/FOSDEM21
- If necessary, create a Pentabarf account and activate it.
Please reuse your account from previous years if you have
already created it.
https://penta.fosdem.org/user/new_account/FOSDEM21
- In the "Person" section, provide First name, Last name
(in the "General" tab), Email (in the "Contact" tab)
and Bio ("Abstract" field in the "Description" tab).
- Submit a proposal by clicking on "Create event".
- If you plan to register your proposal in several tracks to increase your chances,
don't! Register your talk once, in the most accurate track.
- Presentations have to be pre-recorded before the event and will be streamed on
the event weekend.
- Important! Select the "Software Defined Storage devroom" track
(on the "General" tab).
- Provide the title of your talk ("Event title" in the "General" tab).
- Provide a description of the subject of the talk and the
intended audience (in the "Abstract" field of the "Description" tab)
- Provide a rough outline of the talk or goals of the session (a short
list of bullet points covering topics that will be discussed) in the
"Full description" field in the "Description" tab
- Provide an expected length of your talk in the "Duration" field.
We suggest a length between 15 and 45 minutes.
** For accepted talks
Once your proposal is accepted we will assign you a volunteer deputy who will
help you to produce the talk recording. The volunteer will also try to ensure
the recording is of good quality, help with uploading it to the system,
broadcasting it during the event and moderate the Q session after the
broadcast. Please note that as a presenter you're expected to be available
online during and especially after the broadcast of you talk. The schedule will
be available under
https://fosdem.org/2021/schedule/track/software_defined_storage/
Hope to hear from you soon! And please forward this announcement.
If you have any further questions, please write to the mailing list at
storage-devr...@lists.fosdem.org and we will try to answer as soon as
possible.
Thanks!
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] How to copy an OSD from one failing disk to another one
Hi, dear cephers, On one ceph i have a failing disk, whose SMART information signals an impending failure but still availble for reads and writes. I am setting up a new disk on the same node to replace it. What is the best procedure to migrate data (or COPY ) from the failing OSD to the new one? Is there any stardard method to copy the OSD from one to another? best regards, samuel huxia...@horebdata.cn ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: MDS lost, Filesystem degraded and wont mount
Wow! Distributed epins :) Thanks for trying it. How many sub-directories under the distributed epin'd directory? (There's a lot of stability problems that are to be fixed in Pacific associated with lots of subtrees so if you have too large of a directory, things could get ugly!) Yay, beta testing in production! ^^ We are talking millions, but the three is very deep, not very wide. That's why it's so hard to maintain manual pins. I enabled it on a few levels of the tree, where the largest one has 117 direct descendants (but several million files below). So far, it's working all right, but it is very hard to see if the settings is actually effective. I enabled it for testing purposes on a directory that was (at that time) rather busy with 3k MDS op/s and I could see a handful of new pins come and go in ceph tell mds.0 get subtrees, but most of our directories are rather idle most of the time and manually browsing the tree isn't enough to trigger any new observable epins it seems. So for the main directories where it actually matters, I can only assume that it's working. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
