[Bug 1628750] Re: Please backport fixes from 10.2.3 and tip for RadosGW
** Tags removed: verification-mitaka-needed ** Tags added: verification-mitaka-done ** Changed in: cloud-archive/mitaka Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1628750 Title: Please backport fixes from 10.2.3 and tip for RadosGW To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1628750/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1628750] Re: Please backport fixes from 10.2.3 and tip for RadosGW
This bug was fixed in the package ceph - 10.2.3-0ubuntu0.16.04.2
---
ceph (10.2.3-0ubuntu0.16.04.2) xenial; urgency=medium
* rgw: Fixes for creation times for buckets (LP: #1587261):
- d/p/rgw_rados-creation_time.patch: Backport fix from upstream master.
Fix logic error that leads to creation time being 0 instead of current
time when creating buckets.
ceph (10.2.3-0ubuntu0.16.04.1) xenial; urgency=medium
* New upstream stable release (LP: #1628809).
- d/p/*: Refresh.
- d/p/rocksdb-flags.patch: Dropped, accepted upstream.
- d/p/32bit-ftbfs.patch: Cherry pick fix for 32bit arch compat.
- d/ceph-{fs-common,fuse}.install: Fix install locations
for mount{.fuse}.ceph.
* Limit the amount of data per chunk in omap push operations to 64k,
ensuring that OSD threads don't hit timeouts during recovery
operations (LP: #1628750):
- d/p/osd-limit-omap-data-in-push-op.patch: Cherry pick fix from
upstream master branch.
-- Frode Nordahl Fri, 28 Oct 2016
13:50:40 +0200
** Changed in: ceph (Ubuntu Xenial)
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1628750
Title:
Please backport fixes from 10.2.3 and tip for RadosGW
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1628750/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1628750] Re: Please backport fixes from 10.2.3 and tip for RadosGW
** Tags removed: verification-needed ** Tags added: verification-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1628750 Title: Please backport fixes from 10.2.3 and tip for RadosGW To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1628750/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1628750] Re: Please backport fixes from 10.2.3 and tip for RadosGW
** Tags added: sts -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1628750 Title: Please backport fixes from 10.2.3 and tip for RadosGW To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1628750/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1628750] Re: Please backport fixes from 10.2.3 and tip for RadosGW
Working on a lab+test to attempt to verify this on Xenial. Will report progress as soon as I have predictible and recurring results. @elmo: How is the patch holding up in your Trusty+Mitaka environment so far? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1628750 Title: Please backport fixes from 10.2.3 and tip for RadosGW To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1628750/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1628750] Re: Please backport fixes from 10.2.3 and tip for RadosGW
FYI, I can't test the xenial-proposed upload on the customer site as they're running Trusty with Mitaka providing Ceph. I'll try and find a Xenial deployment to test this on, but we don't have many customers using RadosGW. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1628750 Title: Please backport fixes from 10.2.3 and tip for RadosGW To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1628750/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1628750] Re: Please backport fixes from 10.2.3 and tip for RadosGW
Hello James, or anyone else affected, Accepted ceph into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/10.2.3-0ubuntu0.16.04.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: ceph (Ubuntu Xenial) Status: Triaged => Fix Committed ** Tags added: verification-needed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1628750 Title: Please backport fixes from 10.2.3 and tip for RadosGW To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1628750/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1628750] Re: Please backport fixes from 10.2.3 and tip for RadosGW
This bug was fixed in the package ceph - 10.2.3-0ubuntu1
---
ceph (10.2.3-0ubuntu1) yakkety; urgency=medium
* New upstream point release (LP: #1628809):
- d/p/rocksdb-flags.patch: Dropped, included upstream.
- d/p/*: Refreshed.
- d/p/32bit-ftbfs.patch: Cherry pick fix for 32bit arch compat.
- d/ceph-{fs-common,fuse}.install: Fix install locations
for mount{.fuse}.ceph.
* Limit the amount of data per chunk in omap push operations to 64k,
ensuring that OSD threads don't hit timeouts during recovery
operations (LP: #1628750):
- d/p/osd-limit-omap-data-in-push-op.patch: Cherry pick fix from
upstream master branch.
-- James Page Thu, 29 Sep 2016 21:44:33 +0100
** Changed in: ceph (Ubuntu Yakkety)
Status: Triaged => Fix Released
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1628750
Title:
Please backport fixes from 10.2.3 and tip for RadosGW
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1628750/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1628750] Re: Please backport fixes from 10.2.3 and tip for RadosGW
** Description changed: + [Impact] + In ceph deployments with large numbers of objects (typically generated by use of radosgw for object storage), during recovery options when servers or disks fail, it quite possible for OSD recovering data to hit their suicide timeout and shutdown because of the number of objects each was trying to recover in a single chuck between heartbeats. As a result, clusters go read-only due to data availability. + + [Test Case] + Non-trivial to reproduce - see original bug report. + + [Regression Potential] + Medium; the fix for this problem is to reduce the number of operations per chunk to 64000, limiting the chance that an OSD will not heatbeat and suicide itself as a result. This is configurable so can be tuned on a per environment basis. + + The patch has been accepted into the Ceph master branch, but is not + currently targetted as a stable fix for Jewel. + + >> Original Bug Report << + We've run into significant issues with RadosGW at scale; we have a customer who has ½ billion objects in ~20Tb of data and whenever they lose an OSD for whatever reason, even for a very short period of time, ceph was taking hours and hours to recover. The whole time it was recovering requests to RadosGW were hanging. I ended up cherry picking 3 patches; 2 from 10.2.3 and one from trunk: * d/p/fix-pg-temp.patch: cherry pick 56bbcb1aa11a2beb951de396b0de9e3373d91c57 from jewel. * d/p/only-update-up_thru-if-newer.patch: 6554d462059b68ab983c0c8355c465e98ca45440 from jewel. * d/p/limit-omap-data-in-push-op.patch: 38609de1ec5281602d925d20c392ba4094fdf9d3 from master. The 2 from 10.2.3 are because pg_temp was implicated in one of the longer outages we had. The last one is what I think actually got us to a point where ceph was stable and I found it via the following URL chain: http://lists.opennebula.org/pipermail/ceph-users-ceph.com/2016-June/010230.html -> http://tracker.ceph.com/issues/16128 -> https://github.com/ceph/ceph/pull/9894 -> https://github.com/ceph/ceph/commit/38609de1ec5281602d925d20c392ba4094fdf9d3 With these 3 patches applied the customer has been stable for 4 days now but I've yet to restart the entire cluster (only the stuck OSDs) so it's hard to be completely sure that all our issues are resolved but also which of the patches fixed things. I've attached the debdiff I used for reference. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1628750 Title: Please backport fixes from 10.2.3 and tip for RadosGW To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1628750/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1628750] Re: Please backport fixes from 10.2.3 and tip for RadosGW
** Changed in: cloud-archive/mitaka Assignee: (unassigned) => James Page (james-page) ** Changed in: ceph (Ubuntu Xenial) Assignee: (unassigned) => James Page (james-page) ** Changed in: ceph (Ubuntu Yakkety) Assignee: (unassigned) => James Page (james-page) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1628750 Title: Please backport fixes from 10.2.3 and tip for RadosGW To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1628750/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1628750] Re: Please backport fixes from 10.2.3 and tip for RadosGW
10.2.3 tracked under bug 1628809 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1628750 Title: Please backport fixes from 10.2.3 and tip for RadosGW To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1628750/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1628750] Re: Please backport fixes from 10.2.3 and tip for RadosGW
** Changed in: ceph (Ubuntu) Status: New => Triaged ** Changed in: ceph (Ubuntu) Importance: Undecided => Critical ** Also affects: ceph (Ubuntu Yakkety) Importance: Critical Status: Triaged ** Also affects: ceph (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: cloud-archive Importance: Undecided Status: New ** Also affects: cloud-archive/mitaka Importance: Undecided Status: New ** Changed in: cloud-archive Status: New => Invalid ** Changed in: cloud-archive/mitaka Status: New => Triaged ** Changed in: ceph (Ubuntu Xenial) Status: New => Triaged ** Changed in: cloud-archive/mitaka Importance: Undecided => Critical ** Changed in: ceph (Ubuntu Xenial) Importance: Undecided => Critical -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1628750 Title: Please backport fixes from 10.2.3 and tip for RadosGW To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1628750/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1628750] Re: Please backport fixes from 10.2.3 and tip for RadosGW
The attachment "ceph.debdiff" seems to be a debdiff. The ubuntu- sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team. [This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.] ** Tags added: patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1628750 Title: Please backport fixes from 10.2.3 and tip for RadosGW To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1628750/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1628750] Re: Please backport fixes from 10.2.3 and tip for RadosGW
For what it's worth we're running Ubuntu 14.04 + Mitaka from the Ubuntu Cloud Archive. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1628750 Title: Please backport fixes from 10.2.3 and tip for RadosGW To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1628750/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1628750] Re: Please backport fixes from 10.2.3 and tip for RadosGW
** Patch added: "ceph.debdiff" https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1628750/+attachment/4750653/+files/ceph.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1628750 Title: Please backport fixes from 10.2.3 and tip for RadosGW To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1628750/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
