Hi Peter, For the [Test Plan], I added some more clarifying steps as it was a little ambiguous on how this will be tested to ensure the issue is fixed. Please read my changes, and edit if necessary to make it more clear how this will be verified.
If you have a zaza testsuite, please link and attach the output from any successful runs with potential -proposed packages for verification. Thanks, Matthew ** Description changed: [ Impact ] - * The Ceph RADOS gateway is consistently crashing in version 20.2.0, - on Ubuntu resolute, when used keystone for authentication. + * The Ceph RADOS gateway is consistently crashing in version 20.2.0, + on Ubuntu resolute, when used keystone for authentication. - * The new version, 20.2.1, fixes this issue by reverting the behaviour of - the http handler to a way more in line with the implementation in - Squid (19.2.3). + * The new version, 20.2.1, fixes this issue by reverting the behaviour of + the http handler to a way more in line with the implementation in + Squid (19.2.3). [ Test Plan ] - * The issue can be reproduced by running a basic S3 workflow with the - RADOS gateway and using keystone for authentication. + * The issue can be reproduced by running a basic S3 workflow with the + RADOS gateway and using keystone for authentication. - * We will verify that the issue is no longer present by running the functional - test suite for the Ceph charms, which include appropriate tests for RGW. + * This needs to be tested with a similar software stack used in the output from "juju status" in the original description. ceph-mon, ceph-osd, ceph-radosgw + keystone, mysql and vault. + + * When exercising the S3 workflow, the crash needs to be demonstrated + on 20.2.0, and then showed it being fixed in 20.2.4. + + * If an automated zaza test is available, this must be linked to the + comments in an attachment, along with a copy of the output as an + attachment. [ Where problems could occur ] - * The new version reverts part of the behaviour in the HTTP handling of the - RADOS gateway. Since this is an integral part of the gateway, any issues - that are introduced by these changes should be immediately visible. + * The new version reverts part of the behaviour in the HTTP handling of the + RADOS gateway. Since this is an integral part of the gateway, any issues + that are introduced by these changes should be immediately visible. - * The new version uses the same implementation as what it was in the previous - packages (19.2.3). No issues with RADOS gateway were found with them. + * The new version uses the same implementation as what it was in the previous + packages (19.2.3). No issues with RADOS gateway were found with them. [ Original Description ] On Ubuntu resolute radosgw from 20.2.0-0ubuntu2 reproducibly segfaults when an S3 request is authenticated with Keystone EC2 credentials. The client sees "502 Bad Gateway" and systemd restarts radosgw. This is the Keystone-backed S3 auth path. Local RGW users can work when local auth is tried first, but a request using Keystone EC2 credentials still crashes RGW. This can be reproduced using the CI bundle with adaptation for resolute and zaza tests Stack trace ``` Thread 602 "http_manager" received signal SIGSEGV, Segmentation fault. 0x00005e52632bf3b6 in ceph::async::detail::CompletionImpl<boost::asio::any_io_executor, boost::asio::detail::spawn_handler<boost::asio::any_io_executor, void (boost::system::error_code), void>, void, boost::system::error_code>::destroy_post(std::tuple<boost::system::error_code>&&) () Thread 602 (Thread 0x753657aea6c0 (LWP 63226) "http_manager"): #0 0x00005e52632bf3b6 in ceph::async::detail::CompletionImpl<boost::asio::any_io_executor, boost::asio::detail::spawn_handler<boost::asio::any_io_executor, void (boost::system::error_code), void>, void, boost::system::error_code>::destroy_post(std::tuple<boost::system::error_code>&&) () #1 0x00005e5263345d18 in RGWHTTPManager::finish_request(rgw_http_req_data*, int, long) () #2 0x00005e5263346c5a in RGWHTTPManager::reqs_thread_entry() () #3 0x00005e52633475e1 in RGWHTTPManager::ReqsThread::entry() () #4 0x000075365eca40da in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:454 #5 0x000075365ed377ac in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 ``` Juju status ``` # juju status Model Controller Cloud/Region Version SLA Timestamp zaza-9bef958d671a lxd localhost/localhost 3.6.23 unsupported 11:39:43Z App Version Status Scale Charm Channel Rev Exposed Message ceph-mon 20.2.0 active 3 ceph-mon 0 no Unit is ready and clustered ceph-osd 20.2.0 active 3 ceph-osd 15 no Unit is ready (1 OSD) ceph-radosgw 20.2.0 active 1 ceph-radosgw 26 no Unit is ready keystone 25.0.0 active 1 keystone latest/edge 774 no Application Ready mysql 8.0.44-0ubun... active 1 mysql 8.0/stable 444 no vault 1.8.8 active 1 vault latest/edge 363 no Unit is ready (active: true, mlock: disabled) Unit Workload Agent Machine Public address Ports Message ceph-mon/0* active idle 4 10.33.157.168 Unit is ready and clustered ceph-mon/1 active idle 5 10.33.157.25 Unit is ready and clustered ceph-mon/2 active idle 6 fd42:34da:d2af:4f10:216:3eff:fea3:1a7b Unit is ready and clustered ceph-osd/0* active idle 1 fd42:34da:d2af:4f10:216:3eff:fe5d:8e95 Unit is ready (1 OSD) ceph-osd/1 active idle 2 10.33.157.197 Unit is ready (1 OSD) ceph-osd/2 active idle 3 10.33.157.131 Unit is ready (1 OSD) ceph-radosgw/0* active idle 0 fd42:34da:d2af:4f10:216:3eff:fece:c073 80/tcp Unit is ready keystone/0* active idle 8 10.33.157.200 5000/tcp Unit is ready mysql/0* active idle 7 10.33.157.76 3306,33060/tcp Primary vault/0* active idle 9 fd42:34da:d2af:4f10:216:3eff:fe01:376d 8200/tcp Unit is ready (active: true, mlock: disabled) Machine State Address Inst id Base AZ Message 0 started fd42:34da:d2af:4f10:216:3eff:fece:c073 juju-a24339-0 [email protected] sunkern Running 1 started fd42:34da:d2af:4f10:216:3eff:fe5d:8e95 juju-a24339-1 [email protected] sunkern Running 2 started 10.33.157.197 juju-a24339-2 [email protected] sunkern Running 3 started 10.33.157.131 juju-a24339-3 [email protected] sunkern Running 4 started 10.33.157.168 juju-a24339-4 [email protected] sunkern Running 5 started 10.33.157.25 juju-a24339-5 [email protected] sunkern Running 6 started fd42:34da:d2af:4f10:216:3eff:fea3:1a7b juju-a24339-6 [email protected] sunkern Running 7 started 10.33.157.76 juju-a24339-7 [email protected] sunkern Running 8 started 10.33.157.200 juju-a24339-8 [email protected] sunkern Running 9 started fd42:34da:d2af:4f10:216:3eff:fe01:376d juju-a24339-9 [email protected] sunkern Running ``` versions ``` Ubuntu 26.04 LTS (resolute) radosgw: 20.2.0-0ubuntu2 ceph-common: 20.2.0-0ubuntu2 librados2: 20.2.0-0ubuntu2 radosgw --version: - ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) + ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) ``` -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2154304 Title: ceph radosgw segv with keystone auth To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2154304/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
