Also reported upstream: https://tracker.ceph.com/issues/37386
** Description changed: + [Impact] + radosgw is non functional - files cannot be uploaded. + + [Test Case] + Deploy ceph + radosgw + Upload file using swift cli tooling + radosgw daemon SIGABRT's. + + [Regression Potential] + The cause of this bug is a distro patch for 32bit compatibility which mis-aligns the type for the retrieval of a configuration option with the underlying type of the option - this results in a SIGABRT. + + The update to the distro patch realigns the types and aligns with a + broader patch in upstream ceph in master branch development. + + [Original Bug Report] Hi, Juju deployed OpenStack using a modified openstack-core bundle - Rocky on bionic. All components (except neutron-gateway) are deployed in LXD containers. Horizon/CLI can list and create new containers, however, if a small file (in this case a cloud.init file of 82 bytes) is uploaded, the radosgw process crashes with attached crash file. NB this is running within GCP Please advise if any further information required. Here's the snippet out of the end of radosgw.log d9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e80) v8 -- 0x5587e73f9740 con 0 - -27> 2018-11-26 12:17:31.262 7f87d596b700 5 -- 252.0.89.179:0/1413084365 >> 252.0.80.134:6800/7301 conn(0x5587e6c51800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.0 seq 10832 0x5587e73f9740 osd_op_reply(132197 ad9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 - -26> 2018-11-26 12:17:31.262 7f87d596b700 1 -- 252.0.89.179:0/1413084365 <== osd.0 252.0.80.134:6800/7301 10832 ==== osd_op_reply(132197 ad9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 251+0+0 (2451902153 0 0) 0x5587e73f9740 con 0x5587e6c51800 - -25> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001665:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:normalizing buckets and tenants - -24> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001701:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:init permissions - -23> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001755:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:recalculating target - -22> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001779:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:reading permissions - -21> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001786:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:init op - -20> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001788:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:verifying op mask - -19> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001789:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:verifying op permissions - -18> 2018-11-26 12:17:31.262 7f87b633a700 5 content type wasn't provided, trying to guess - -17> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=ad9fa6cacbb94b96b4941ec2af38a49f, acct_name=xxxx_mgmt, perm_mask=15, is_admin=0) mask=50 - -16> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for uid=ad9fa6cacbb94b96b4941ec2af38a49f - -15> 2018-11-26 12:17:31.262 7f87b633a700 5 Found permission: 15 - -14> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for uid=ad9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f - -13> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for user not found - -12> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for group=1 mask=50 - -11> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for group not found - -10> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for group=2 mask=50 - -9> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for group not found - -8> 2018-11-26 12:17:31.262 7f87b633a700 5 -- Getting permissions done for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=ad9fa6cacbb94b96b4941ec2af38a49f, acct_name=xxxxx_mgmt, perm_mask=15, is_admin=0), owner=ad9fa6cacbb94b96b4941ec2af38a49f, perm=2 - -7> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001818:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:verifying op params - -6> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001820:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:pre-executing - -5> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001821:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:executing - -4> 2018-11-26 12:17:31.262 7f87b633a700 1 -- 252.0.89.179:0/1413084365 --> 252.0.90.230:6800/17596 -- osd_op(unknown.0.0:132198 6.0 6:21b9d9f7:::.dir.c9c6424e-a4bd-44de-b972-6cd4539a0a61.5322.1:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e80) v8 -- 0x5587e745a9c0 con 0 - -3> 2018-11-26 12:17:31.266 7f87d616c700 5 -- 252.0.89.179:0/1413084365 >> 252.0.90.230:6800/17596 conn(0x5587e6c50300 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=10 cs=1 l=1). rx osd.1 seq 54313 0x5587e745a9c0 osd_op_reply(132198 .dir.c9c6424e-a4bd-44de-b972-6cd4539a0a61.5322.1 [call] v0'0 uv1 ondisk = 0) v8 - -2> 2018-11-26 12:17:31.266 7f87d616c700 1 -- 252.0.89.179:0/1413084365 <== osd.1 252.0.90.230:6800/17596 54313 ==== osd_op_reply(132198 .dir.c9c6424e-a4bd-44de-b972-6cd4539a0a61.5322.1 [call] v0'0 uv1 ondisk = 0) v8 ==== 192+0+71 (1074935246 0 1943469035) 0x5587e745a9c0 con 0x5587e6c50300 - -1> 2018-11-26 12:17:31.266 7f87b633a700 5 NOTICE: call to do_aws4_auth_completion - 0> 2018-11-26 12:17:31.266 7f87b633a700 -1 *** Caught signal (Aborted) ** - in thread 7f87b633a700 thread_name:civetweb-worker + -27> 2018-11-26 12:17:31.262 7f87d596b700 5 -- 252.0.89.179:0/1413084365 >> 252.0.80.134:6800/7301 conn(0x5587e6c51800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.0 seq 10832 0x5587e73f9740 osd_op_reply(132197 ad9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 + -26> 2018-11-26 12:17:31.262 7f87d596b700 1 -- 252.0.89.179:0/1413084365 <== osd.0 252.0.80.134:6800/7301 10832 ==== osd_op_reply(132197 ad9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 251+0+0 (2451902153 0 0) 0x5587e73f9740 con 0x5587e6c51800 + -25> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001665:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:normalizing buckets and tenants + -24> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001701:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:init permissions + -23> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001755:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:recalculating target + -22> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001779:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:reading permissions + -21> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001786:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:init op + -20> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001788:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:verifying op mask + -19> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001789:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:verifying op permissions + -18> 2018-11-26 12:17:31.262 7f87b633a700 5 content type wasn't provided, trying to guess + -17> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=ad9fa6cacbb94b96b4941ec2af38a49f, acct_name=xxxx_mgmt, perm_mask=15, is_admin=0) mask=50 + -16> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for uid=ad9fa6cacbb94b96b4941ec2af38a49f + -15> 2018-11-26 12:17:31.262 7f87b633a700 5 Found permission: 15 + -14> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for uid=ad9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f + -13> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for user not found + -12> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for group=1 mask=50 + -11> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for group not found + -10> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for group=2 mask=50 + -9> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for group not found + -8> 2018-11-26 12:17:31.262 7f87b633a700 5 -- Getting permissions done for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=ad9fa6cacbb94b96b4941ec2af38a49f, acct_name=xxxxx_mgmt, perm_mask=15, is_admin=0), owner=ad9fa6cacbb94b96b4941ec2af38a49f, perm=2 + -7> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001818:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:verifying op params + -6> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001820:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:pre-executing + -5> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001821:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:executing + -4> 2018-11-26 12:17:31.262 7f87b633a700 1 -- 252.0.89.179:0/1413084365 --> 252.0.90.230:6800/17596 -- osd_op(unknown.0.0:132198 6.0 6:21b9d9f7:::.dir.c9c6424e-a4bd-44de-b972-6cd4539a0a61.5322.1:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e80) v8 -- 0x5587e745a9c0 con 0 + -3> 2018-11-26 12:17:31.266 7f87d616c700 5 -- 252.0.89.179:0/1413084365 >> 252.0.90.230:6800/17596 conn(0x5587e6c50300 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=10 cs=1 l=1). rx osd.1 seq 54313 0x5587e745a9c0 osd_op_reply(132198 .dir.c9c6424e-a4bd-44de-b972-6cd4539a0a61.5322.1 [call] v0'0 uv1 ondisk = 0) v8 + -2> 2018-11-26 12:17:31.266 7f87d616c700 1 -- 252.0.89.179:0/1413084365 <== osd.1 252.0.90.230:6800/17596 54313 ==== osd_op_reply(132198 .dir.c9c6424e-a4bd-44de-b972-6cd4539a0a61.5322.1 [call] v0'0 uv1 ondisk = 0) v8 ==== 192+0+71 (1074935246 0 1943469035) 0x5587e745a9c0 con 0x5587e6c50300 + -1> 2018-11-26 12:17:31.266 7f87b633a700 5 NOTICE: call to do_aws4_auth_completion + 0> 2018-11-26 12:17:31.266 7f87b633a700 -1 *** Caught signal (Aborted) ** + in thread 7f87b633a700 thread_name:civetweb-worker - ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable) - 1: (()+0x293e70) [0x5587e4514e70] - 2: (()+0x12890) [0x7f87eb99d890] - 3: (gsignal()+0xc7) [0x7f87e0e96e97] - 4: (abort()+0x141) [0x7f87e0e98801] - 5: (()+0x8c8b7) [0x7f87e14ed8b7] - 6: (()+0x92a06) [0x7f87e14f3a06] - 7: (()+0x92a41) [0x7f87e14f3a41] - 8: (()+0x92c74) [0x7f87e14f3c74] - 9: (()+0x35e30d) [0x5587e45df30d] - 10: (Option::size_t const md_config_t::get_val<Option::size_t>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const+0x53) [0x5587e461fab3] - 11: (()+0x37df4f) [0x5587e45fef4f] - 12: (RGWPutObj::execute()+0x1adb) [0x5587e4600efb] - 13: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x1cd) [0x5587e4620cfd] - 14: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, int*)+0x15f0) [0x5587e46223f0] - 15: (RGWCivetWebFrontend::process(mg_connection*)+0x381) [0x5587e44976e1] - 16: (()+0x2851ee) [0x5587e45061ee] - 17: (()+0x286e78) [0x5587e4507e78] - 18: (()+0x287308) [0x5587e4508308] - 19: (()+0x76db) [0x7f87eb9926db] - 20: (clone()+0x3f) [0x7f87e0f7988f] - NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. + ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable) + 1: (()+0x293e70) [0x5587e4514e70] + 2: (()+0x12890) [0x7f87eb99d890] + 3: (gsignal()+0xc7) [0x7f87e0e96e97] + 4: (abort()+0x141) [0x7f87e0e98801] + 5: (()+0x8c8b7) [0x7f87e14ed8b7] + 6: (()+0x92a06) [0x7f87e14f3a06] + 7: (()+0x92a41) [0x7f87e14f3a41] + 8: (()+0x92c74) [0x7f87e14f3c74] + 9: (()+0x35e30d) [0x5587e45df30d] + 10: (Option::size_t const md_config_t::get_val<Option::size_t>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const+0x53) [0x5587e461fab3] + 11: (()+0x37df4f) [0x5587e45fef4f] + 12: (RGWPutObj::execute()+0x1adb) [0x5587e4600efb] + 13: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x1cd) [0x5587e4620cfd] + 14: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, int*)+0x15f0) [0x5587e46223f0] + 15: (RGWCivetWebFrontend::process(mg_connection*)+0x381) [0x5587e44976e1] + 16: (()+0x2851ee) [0x5587e45061ee] + 17: (()+0x286e78) [0x5587e4507e78] + 18: (()+0x287308) [0x5587e4508308] + 19: (()+0x76db) [0x7f87eb9926db] + 20: (clone()+0x3f) [0x7f87e0f7988f] + NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- - 0/ 5 none - 0/ 1 lockdep - 0/ 1 context - 1/ 1 crush - 1/ 5 mds - 1/ 5 mds_balancer - 1/ 5 mds_locker - 1/ 5 mds_log - 1/ 5 mds_log_expire - 1/ 5 mds_migrator - 0/ 1 buffer - 0/ 1 timer - 0/ 1 filer - 0/ 1 striper - 0/ 1 objecter - 0/ 5 rados - 0/ 5 rbd - 0/ 5 rbd_mirror - 0/ 5 rbd_replay - 0/ 5 journaler - 0/ 5 objectcacher - 0/ 5 client - 1/ 5 osd - 0/ 5 optracker - 0/ 5 objclass - 1/ 3 filestore - 1/ 3 journal - 0/ 5 ms - 1/ 5 mon - 0/10 monc - 1/ 5 paxos - 0/ 5 tp - 1/ 5 auth - 1/ 5 crypto - 1/ 1 finisher - 1/ 1 reserver - 1/ 5 heartbeatmap - 1/ 5 perfcounter - 1/ 5 rgw - 1/ 5 rgw_sync - 1/10 civetweb - 1/ 5 javaclient - 1/ 5 asok - 1/ 1 throttle - 0/ 0 refs - 1/ 5 xio - 1/ 5 compressor - 1/ 5 bluestore - 1/ 5 bluefs - 1/ 3 bdev - 1/ 5 kstore - 4/ 5 rocksdb - 4/ 5 leveldb - 4/ 5 memdb - 1/ 5 kinetic - 1/ 5 fuse - 1/ 5 mgr - 1/ 5 mgrc - 1/ 5 dpdk - 1/ 5 eventtrace - -2/-2 (syslog threshold) - -1/-1 (stderr threshold) - max_recent 10000 - max_new 1000 - log_file /var/log/ceph/radosgw.log + 0/ 5 none + 0/ 1 lockdep + 0/ 1 context + 1/ 1 crush + 1/ 5 mds + 1/ 5 mds_balancer + 1/ 5 mds_locker + 1/ 5 mds_log + 1/ 5 mds_log_expire + 1/ 5 mds_migrator + 0/ 1 buffer + 0/ 1 timer + 0/ 1 filer + 0/ 1 striper + 0/ 1 objecter + 0/ 5 rados + 0/ 5 rbd + 0/ 5 rbd_mirror + 0/ 5 rbd_replay + 0/ 5 journaler + 0/ 5 objectcacher + 0/ 5 client + 1/ 5 osd + 0/ 5 optracker + 0/ 5 objclass + 1/ 3 filestore + 1/ 3 journal + 0/ 5 ms + 1/ 5 mon + 0/10 monc + 1/ 5 paxos + 0/ 5 tp + 1/ 5 auth + 1/ 5 crypto + 1/ 1 finisher + 1/ 1 reserver + 1/ 5 heartbeatmap + 1/ 5 perfcounter + 1/ 5 rgw + 1/ 5 rgw_sync + 1/10 civetweb + 1/ 5 javaclient + 1/ 5 asok + 1/ 1 throttle + 0/ 0 refs + 1/ 5 xio + 1/ 5 compressor + 1/ 5 bluestore + 1/ 5 bluefs + 1/ 3 bdev + 1/ 5 kstore + 4/ 5 rocksdb + 4/ 5 leveldb + 4/ 5 memdb + 1/ 5 kinetic + 1/ 5 fuse + 1/ 5 mgr + 1/ 5 mgrc + 1/ 5 dpdk + 1/ 5 eventtrace + -2/-2 (syslog threshold) + -1/-1 (stderr threshold) + max_recent 10000 + max_new 1000 + log_file /var/log/ceph/radosgw.log --- end dump of recent events --- ProblemType: Crash DistroRelease: Ubuntu 18.04 Package: radosgw 13.2.1+dfsg1-0ubuntu2~cloud0 [origin: Canonical] ProcVersionSignature: Ubuntu 4.15.0-1024.25-gcp 4.15.18 Uname: Linux 4.15.0-1024-gcp x86_64 ApportVersion: 2.20.9-0ubuntu7.5 Architecture: amd64 CrashDB: - { - "impl": "launchpad", - "project": "cloud-archive", - "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml", - } + { + "impl": "launchpad", + "project": "cloud-archive", + "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml", + } Date: Mon Nov 26 12:17:31 2018 ExecutablePath: /usr/bin/radosgw ProcCmdline: /usr/bin/radosgw -n client.radosgw.gateway ProcEnviron: - PATH=(custom, no user) - LANG=C.UTF-8 + PATH=(custom, no user) + LANG=C.UTF-8 Signal: 6 SourcePackage: ceph StacktraceTop: - __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 - __GI_abort () at abort.c:79 - ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 - ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 - std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 + __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 + __GI_abort () at abort.c:79 + ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 + ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 + std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 Title: radosgw crashed with SIGABRT in __GI_raise() UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1805145 Title: radosgw crashed with SIGABRT in __GI_raise() when uploading small file via Horizon/CLI To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1805145/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs